Pith. sign in

REVIEW 3 major objections 6 minor 85 references

NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensembling

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read NeuroTrails claims a shared sparse backbone with dynamically evolving heads can beat full dense ensembles at lower compute.

desk verdict A useful empirical recipe—shared backbone plus dynamic sparse heads—whose headline claim about dynamic topology is real but undersupported by the ablations as reported. read the letter →

arxiv 2505.17909 v1 pith:6MTMXO4Y submitted 2025-05-23 cs.LG cs.AI

Bram Grooten , Farid Hasanov , Chenxiang Zhang , Qiao Xiao , Boqian Wu , Zahra Atashgahi , Ghada Sokar , Shiwei Liu
show 4 more authors
Lu Yin Elena Mocanu Mykola Pechenizkiy Decebal Constantin Mocanu
This is my paper
classification cs.LGcs.AI
keywords dynamicsparsetrainingmodelensemblesmulti-headarchitectureneuraltrailspredictiondisagreementGoldilockszoneefficientinferencenetworks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NeuroTrails tries to establish that you do not need independently trained dense networks to get ensemble-level gains: a single network split into a shared early backbone and several sparse heads, with the sparse topology updated during training, can outperform a full ensemble while using fewer parameters and fewer inference FLOPs. The paper shows this on convolutional networks (ResNet-50 on ImageNet, Wide-ResNet28-10 on CIFAR-100) and transformer language models (LLaMA-130M/350M on C4), with higher ImageNet accuracy and lower C4 perplexity than a full ensemble. The central reason offered is that dynamic sparsity carves out distinct 'neural trails' whose prediction disagreement lands in a Goldilocks zone: enough diversity to help, not so much that heads cancel each other. A reader should care because, if right, ensembling becomes a cheap add-on instead of a multiplied training and inference bill.

What carries the argument

The load-bearing object is a network split at block index $\ell$ into a shared backbone $F_s$ and $M$ independently initialized sparse heads $F_h^{(i)}$, each with its own binary mask. During training the heads' losses are averaged, and every $\Delta T$ steps each mask is updated by dynamic sparse training: a fraction of active weights is pruned by magnitude and the same number of inactive weights is regrown, in the vision experiments by gradient-based regrowth. This constant-density evolution of topology is what the authors call the formation of 'neural trails'—long-range sparse connectivity paths that differ between heads. The mechanisms named as responsible are the shared-backbone split (early features need not be duplicated) and dynamic sparse training (topology exploration creates diversity), with the amount of diversity measured by prediction disagreement and the best regime called the PD Goldilocks zone.

What would settle it

A concrete falsifying experiment is to train NeuroTrails with masks frozen after initialization, keeping head initialization, sparsity, and backbone length identical; if the frozen-mask model matches NeuroTrails in accuracy and robustness, dynamic topology evolution is not the key mechanism. Alternatively, artificially increase prediction disagreement at a fixed architecture, for example with a diversity penalty, and show that accuracy is insensitive to crossing the Goldilocks threshold; if it is, the mechanism is unsupported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that 'comprehensive ensembling of all layers is not a necessary condition to achieve optimal performance.' Sharing early feature extraction through a common backbone and letting each of $M$ sparse heads evolve its own topology through dynamic sparse training yields a model that beats both a single dense network and a full dense ensemble: 78.1% versus 77.5% ImageNet accuracy and 20.7 versus 21.3 C4 perplexity, with inference FLOPs a fraction of the ensemble's. The authors further report stronger robustness to corrupted and out-of-domain images and higher average zero-shot accuracy across seven language tasks. They interpret the gains through prediction disagreement: accuracy peaks when inter-head disagreement is around 14.6%, neither too low (no ensemble benefit) nor too high (heads compete), and they show that too much head-specific depth pushes disagreement past this sweet spot.

Load-bearing premise

The load-bearing premise is that dynamic sparse training, rather than the shared-backbone split or sparsity alone, is what generates the just-right prediction diversity that produces the gains; the paper varies backbone length and observes disagreement, but never varies disagreement while holding the architecture fixed.

Editorial extensions

If this is right

  • Ensembling no longer requires training several independent dense networks; a single multi-head sparse model with the same training budget can beat the ensemble.
  • The shared backbone means inference FLOPs grow much more slowly with ensemble size, and with sparsity-aware software the throughput approaches that of a single dense model.
  • The recipe transfers across architectures and modalities, so the same split-and-evolve scheme can be applied to new convolutional or transformer backbones.
  • Prediction-disagreement monitoring gives a practical design target: choose the backbone and head split so disagreement stays in the Goldilocks zone rather than maximizing diversity.
  • On a fixed parameter budget, NeuroTrails can beat a larger dense model (LLaMA-250M), suggesting the method's gains are not just added capacity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A controlled ablation that holds architecture, sparsity, and head initialization fixed and only toggles the dynamic topology updates on or off would isolate the mechanism, which the current analysis does not directly vary.
  • The same structure could be probed in other settings where prediction diversity is known to matter, such as continual learning or out-of-distribution detection; NeuroTrails predicts a Goldilocks disagreement also governs those regimes.
  • For language models the reported gains use only 10% sparsity, so the method's advantage there may come mostly from head separation rather than sparsity, and sweeping sparsity for LLaMA would separate these factors.
  • If the Goldilocks zone is a general phenomenon, prediction disagreement could be used as an early-stopping or split-selection criterion during training rather than evaluated only after the fact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes NeuroTrails, an ensembling method that splits a base network into a shared backbone and multiple independently initialized sparse heads, then trains the whole multi-head model with dynamic sparse training (RigL or SET). Predictions are combined by soft voting at inference. The authors report that NeuroTrails outperforms a full ensemble and TreeNet on CIFAR-100, ImageNet, Tiny-ImageNet, and C4 language modeling, while requiring fewer parameters and lower inference FLOPs, and that it shows better zero-shot robustness on ImageNet-C, ImageNet-Sketch, and several downstream language tasks. The paper attributes these gains to dynamic sparse training producing 'neural trails' whose prediction disagreement falls in a 'Goldilocks zone', with Section 5.2 and Figure 3 as the main evidence.

Significance. If the reported results hold, NeuroTrails is a practically useful and model-agnostic recipe for efficient ensembling: it combines TreeNet-style layer sharing with dynamic sparse training, and the paper provides open-source code, detailed hyperparameters, and careful FLOPs accounting across vision and language benchmarks. The consistent gains over full ensembles and the inference-cost reductions are valuable. However, the central explanatory claim—that dynamic topology evolution, rather than sparsity, separate head initialization, or an extended training budget, is the key ingredient—is not yet supported by the controlled evidence, so the significance of the mechanism, as opposed to the recipe, remains uncertain.

major comments (3)
  1. [Section 5.1 / Appendix E.3 / Figure 3] The comparison that supports the 'dynamic sparse heads as key' claim is not schedule-matched. Section E.3 states that sparse variants receive extra training by at most 1/(1−S), and Table 11 reports 450 epochs for NeuroTrails on CIFAR-100, but the training schedules for the 'Static Sparsity' and 'Pruning' baselines in Figure 3 are not reported in Tables 11–14 or in Section E.2. If those baselines were trained for only the dense 250-epoch schedule, the superiority of RigL/SET in Figure 3 could reflect an additional training budget rather than dynamic topology. Please report the exact epoch counts for every method in Figure 3, and ideally run static-sparse and one-shot-pruned baselines under the same 450-epoch budget with the same per-epoch FLOPs, before drawing the conclusion that dynamic topology is the causal factor.
  2. [Section 5.2 / Table 6 / Figure 5] The Goldilocks-zone explanation is a post hoc correlation, not a controlled result. Table 6 varies the number of blocks in the head, which simultaneously changes architecture capacity, parameter count, and the efficiency objective used to select the split point; it never manipulates prediction disagreement while holding the architecture fixed. The PD-accuracy pattern is non-monotonic (e.g., |Fh|=4→6 gives PD 11.2→12.4 with accuracy 82.85→82.71, while |Fh|=10→12 gives PD 15.3→16.0 with accuracy 83.47→83.59), and Table 6 reports no error bars even though Section 5 states that all analysis results are mean and standard deviation over 3 seeds. Appendix K explicitly concedes that alternative factors could contribute. At minimum, the authors should report error bars, test the significance of the accuracy differences around the peak, and run a controlled intervention on diversity (e.g., a diversity regularizer at a fixed |Fh|) before claiming that the PD level is causal.
  3. [Tables 2, 3, 4, and 5] The headline empirical tables lack error bars or seed counts, which matters because several reported gains are small. For example, Table 5 shows an average zero-shot accuracy of 40.45 for NeuroTrails versus 39.96 for TreeNet; Table 16 shows Tiny-ImageNet accuracy of 70.7 for NeuroTrails (M=3, S=0.8) versus 70.8 for Full Ensemble; and Table 15 shows a perplexity gap of 26.48 versus 26.59. Without uncertainty estimates, the reader cannot judge whether these differences are meaningful or within seed noise. Please provide standard deviations or confidence intervals for the headline comparisons, or clearly state that these are single runs.
minor comments (6)
  1. [Abstract / Table 1] The phrase 'reducing the required resources' should specify that the reduction is in parameters and inference FLOPs; training FLOPs in Tables 3 and 4 are 1.10x and 2.27x that of a single dense model, so the claim is not true for training compute relative to a single network.
  2. [Section E.3 / Tables 11-14] The statement that exact training schedules appear in Tables 11–14 is incomplete: the static-sparse and pruning baselines used in Figure 3 are absent from those tables.
  3. [Table 6] Please add the standard deviations promised in Section 5; the PD and accuracy values in Table 6 currently appear to be point estimates.
  4. [Section 5.4 / Table 7] The throughput numbers are reported without stating how many CPU cores or what hardware configuration was used; this would help interpret the speedup claims.
  5. [Appendix K] The sentence conceding that 'alternative factors could also contribute' should be moved into Section 5.2 so the main text does not overstate the Goldilocks-zone mechanism.
  6. [Appendix K] There is a typo: 'conciness' should be 'conciseness'. Also, Figure 2's caption uses 'FLOPS' where the rest of the paper uses 'FLOPs'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: NeuroTrails is benchmarked against external datasets, and its Goldilocks diversity analysis is correlational rather than constructed from the target.

full rationale

The paper's central empirical claims—higher ImageNet accuracy, lower C4 perplexity, and improved zero-shot robustness—are evaluated on external benchmarks (ImageNet, C4, CIFAR-100, Tiny-ImageNet, and downstream language tasks) against dense ensembles and prior efficient-ensemble baselines; no parameter is fitted to these targets and then reported as a prediction of the same quantity. The split point and sparsity ratio are selected on CIFAR-100 and transferred to other tasks, which is standard model selection rather than a self-definitional prediction. The Goldilocks-zone analysis (Table 6 and Figure 5) is an observed correlation between prediction disagreement and accuracy; the paper does not derive the optimal disagreement from the accuracy numbers by construction, and Appendix K explicitly concedes that "alternative factors could also contribute," which weakens causal attribution without making the reasoning circular. Self-citations (SET, ER, the 1/(1−S) schedule observation, and DST ensembles) refer to prior published, independently evaluable work and do not assume NeuroTrails' conclusions. Potential training-schedule differences in Figure 3 are a confound for the method-comparison claim, not a circular reduction. Overall, the derivation chain is self-contained.

Assumptions & free parameters 6 free parameters · 6 assumptions · 2 invented entities

The paper relies on standard assumptions about gradient-based optimization and benchmark validity. The specific risk areas are the prediction disagreement diversity hypothesis, the FLOPs scaling convention, and the transfer of hyperparameters from CIFAR-100 to other tasks. Free parameters are mostly standard hyperparameters, but the sparsity ratio and backbone split point are tuned on the main benchmarks.

free parameters (6)
  • target sparsity ratio S = 0.8 (CIFAR-100), 0.7 (ImageNet), 0.1 (C4), varied in ablations
    Chosen per dataset from validation accuracy; central to the parameter and FLOP reductions claimed. Section 4 and Appendix I report an accuracy peak around S=0.8 on CIFAR-100.
  • number of heads M = 3 for most experiments; 5 for some CIFAR-100 and Tiny-ImageNet runs
    Ensemble size is a design choice; Figure 6 shows diminishing returns after M=3, and larger M is paired with higher sparsity in some runs.
  • backbone split index (blocks in head) = 8 of 12 blocks for Wide-ResNet28-10; 10 of 16 for ResNet-50; 8/16 of 12/24 transformer blocks for LLaMA-130M/350M
    Selected from the backbone length sweep on CIFAR-100 (Section 5.1, Figure 3) and then transferred to other architectures.
  • topology update interval Delta T = 100 (CIFAR-100), 1000 (ImageNet), 50 (language modeling)
    Hyperparameter controlling how often dynamic sparse training prunes and regrows connections; set by convention from the sparse training literature.
  • initial drop fraction p = 0.5, cosine decayed to 0
    Fraction of weights pruned and regrown at each topology update; chosen to encourage topology exploration during training.
  • soft magnitude temperature = 3.0
    Language modeling uses soft magnitude pruning with temperature 3.0 (Table 10), following Zhang et al. (2025).
assumptions (6)
  • domain assumption Randomly initialized independent heads with independent sparse masks remain sufficiently diverse when trained with a joint averaged loss.
    The method depends on heads not collapsing to identical predictions. Table 6 reports prediction disagreement values, but there is no direct check for learner collusion, a known failure mode for jointly trained ensembles.
  • domain assumption Prediction Disagreement is an adequate measure of the diversity that controls ensemble quality.
    Section 5.2 uses PD as the basis for the Goldilocks zone conclusion; other diversity metrics exist (Kuncheva and Whitaker, 2003) and are not evaluated.
  • domain assumption Dynamic sparse training discovers topologies that are at least as good as fixed sparse or pruned topologies for the architectures tested.
    Figure 3 supports this empirically on CIFAR-100, but the same conclusion is assumed to transfer to ImageNet and language models without a dedicated ablation on those datasets.
  • domain assumption FLOPs scale linearly with the fraction of active parameters, with the backward pass costing approximately twice the forward pass.
    All training and inference FLOPs comparisons use the RigL scaling convention (Appendix F); actual runtime depends on hardware sparsity support, as shown in Section J.
  • domain assumption Baseline numbers taken from prior papers are comparable despite different training schedules and implementations.
    Tables 2 and 3 mix results from Havasi et al. (2021), Liu et al. (2022), Lee and Lee (2024), and the authors' own runs; the paper itself notes schedule differences and calls for consistent protocols.
  • standard math Standard supervised learning assumptions: i.i.d. data, standard train/test splits, and public benchmarks are valid proxies for generalization.
    Evaluation uses CIFAR-100, ImageNet, Tiny-ImageNet, and C4; distribution shift is only tested through ImageNet-C, ImageNet-Sketch, and downstream language tasks.
invented entities (2)
  • neural trails
    purpose: Explanatory construct for the diverse sparse connectivity paths formed by dynamic sparse training in each head; claimed to be the source of ensemble diversity.
    The paper observes topology changes and prediction disagreement, but it does not provide an external falsifiable prediction tied specifically to 'trails' that is distinct from the sparsity masks themselves.
  • PD Goldilocks zone
    purpose: Claimed optimal range of prediction disagreement where ensemble accuracy is maximized; used to explain why the 8-block-head configuration performs best.
    The zone is inferred post hoc from Table 6 and a few illustrative examples; no independent experiment targets a predicted disagreement value.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensembling." pith.science (2026). https://pith.science/paper/6MTMXO4Y

@misc{pith2026250517909,
  author       = {Pith},
  title        = {Pith review of: NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensembling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6MTMXO4Y}},
  note         = {Machine review of arXiv:2505.17909}
}
abstract

Model ensembles have long been a cornerstone for improving generalization and robustness in deep learning. However, their effectiveness often comes at the cost of substantial computational overhead. To address this issue, state-of-the-art methods aim to replicate ensemble-class performance without requiring multiple independently trained networks. Unfortunately, these algorithms often still demand considerable compute at inference. In response to these limitations, we introduce $\textbf{NeuroTrails}$, a sparse multi-head architecture with dynamically evolving topology. This unexplored model-agnostic training paradigm improves ensemble performance while reducing the required resources. We analyze the underlying reason for its effectiveness and observe that the various neural trails induced by dynamic sparsity attain a $\textit{Goldilocks zone}$ of prediction diversity. NeuroTrails displays efficacy with convolutional and transformer-based architectures on computer vision and language tasks. Experiments on ResNet-50/ImageNet, LLaMA-350M/C4, among many others, demonstrate increased accuracy and stronger robustness in zero-shot generalization, while requiring significantly fewer parameters.

Figures

Figures reproduced from arXiv: 2505.17909 by the authors.

Figure 1
Figure 1. Illustration of NeuroTrails. We divide a network into a shared backbone [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Testing zero-shot generalization ability on cor [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Performance of NeuroTrails models with varying [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Example of a CIFAR￾100 test-set image where too much prediction diversity between heads degrades performance. NeuroTrails with 8 blocks in each head seems to get the amount of diversity just right. For more illustrations of pre￾dictions with overly large diversity, see…
Figure 5
Figure 5. Figure 5: Accuracy and Prediction Disagreement throughout training for a NeuroTrails model on CIFAR-100, showing an inverse trend. Prediction Disagreement over Time. We observe in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Effect of the ensemble size on CIFAR-100 with Wide-ResNet28-10. NeuroTrails achieves [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Impact of sparsity ratio on accuracy in a NeuroTrails model with three heads ( [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]
Figure 8
Figure 8. Figure 8: Inference latency comparison across NeuroTrails models with increasing sparsity levels [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]
Figure 9
Figure 9. Figure 9: Direct prediction comparison between NeuroTrails models with 8 and 12 blocks in their [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

85 extracted references · 32 canonical work pages

  1. [1]

    Dual Lottery Ticket Hypothesis

    Yue Bai, Huan Wang, Zhiqiang Tao, Kunpeng Li, and Yun Fu. Dual Lottery Ticket Hypothesis . In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL: https://arxiv.org/abs/2203.04248

    work page Pith review arXiv 2022

  2. [2]

    Deep Rewiring: Training very sparse deep networks

    Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep Rewiring: Training very sparse deep networks . CoRR, abs/1711.05136, 2017. URL: https://arxiv.org/abs/1711.05136

    arXiv 2017

  3. [3]

    H. Beyer. Exploratory Data Analysis . Biometrical Journal, 23 0 (4): 0 413–414, January 1981. ISSN 1521-4036. URL: https://doi.org/10.1002/bimj.4710230408

    work page doi:10.1002/bimj.4710230408 1981

  4. [4]

    Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better

    Sameer Bibikar, Haris Vikalo, Zhangyang Wang, and Xiaohan Chen. Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better . Proceedings of the AAAI Conference on Artificial Intelligence, 36 0 (6): 0 6080--6088, 2022. URL: https://arxiv.org/abs/2112.09824

    arXiv 2022

  5. [5]

    PIQA: Reasoning about Physical Commonsense in Natural Language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about Physical Commonsense in Natural Language . CoRR, abs/1911.11641, 2019. URL: https://arxiv.org/abs/1911.11641

    arXiv 1911

  6. [6]

    Bagging predictors

    Leo Breiman. Bagging predictors . Machine Learning, 24 0 (2): 0 123--140, 8 1996. ISSN 0885-6125. URL: https://link.springer.com/article/10.1007/BF00058655

    doi:10.1007/bf00058655 1996

  7. [7]

    Sparsity Made Easy – Introducing the Cerebras PyTorch Sparsity Library - Cerebras , 2024

    Cerebras. Sparsity Made Easy – Introducing the Cerebras PyTorch Sparsity Library - Cerebras , 2024. URL: https://www.cerebras.ai/blog/sparsity-made-easy-introducing-the-cerebras-pytorch-sparsity-library [Accessed: 25.04.2025]

    2024

  8. [8]

    BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

    Christopher Clark, Kenton Lee, Ming - Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions . CoRR, abs/1905.10044, 2019. URL: https://arxiv.org/abs/1905.10044

    arXiv 1905

Show all 85 references
  1. [9]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge . CoRR, abs/1803.05457, 2018. URL: https://arxiv.org/abs/1803.05457

    2018 arXiv

  2. [10]

    Truly Sparse Neural Networks at Scale

    Selima Curci, Decebal Constantin Mocanu, and Mykola Pechenizkiyi. Truly Sparse Neural Networks at Scale . 2021. URL: https://arxiv.org/abs/2102.01732

    2021 arXiv

  3. [11]

    ImageNet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database . In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248--255. IEEE, 2009. URL: https://ieeexplore.ieee.org/document/5206848

    2009

  4. [12]

    Sparse Networks from Scratch: Faster Training without Losing Performance , 2019

    Tim Dettmers and Luke Zettlemoyer. Sparse Networks from Scratch: Faster Training without Losing Performance , 2019. URL: https://arxiv.org/abs/1907.04840

    2019 arXiv

  5. [13]

    Dietterich

    Thomas G. Dietterich. Ensemble Methods in Machine Learning . In Multiple Classifier Systems, pages 1--15. Springer Berlin Heidelberg, 2000. ISBN 978-3-540-45014-6. URL: https://link.springer.com/chapter/10.1007/3-540-45014-9_1

    doi

  6. [14]

    Rigging the Lottery: Making All Tickets Winners

    Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the Lottery: Making All Tickets Winners . In International Conference on Machine Learning, pages 2943--2952. PMLR, 2020. URL: https://arxiv.org/abs/1911.11134

    2020 arXiv

  7. [15]

    Gradient flow in sparse Neural Networks and how Lottery Tickets win

    Utku Evci, Yani Ioannou, Cem Keskin, and Yann Dauphin. Gradient flow in sparse Neural Networks and how Lottery Tickets win . Proc. Conf. AAAI Artif. Intell., 36 0 (6): 0 6577--6586, June 2022. URL: https://arxiv.org/abs/2010.03533

    2022 arXiv

  8. [16]

    Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity . Journal of Machine Learning Research, 2021. URL: https://arxiv.org/abs/2101.03961

    2021 arXiv

  9. [17]

    Deep Ensembles: A Loss Landscape Perspective , 2020

    Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep Ensembles: A Loss Landscape Perspective , 2020. URL: https://arxiv.org/abs/1912.02757

    2020 arXiv

  10. [18]

    The Lottery Ticket Hypothesis: Training Pruned Neural Networks

    Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Training Pruned Neural Networks . International Conference on Learning Representations, 2019. URL: https://arxiv.org/abs/1803.03635

    2019 arXiv

  11. [19]

    A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting

    Yoav Freund and Robert E Schapire. A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting . Journal of Computer and System Sciences, 55 0 (1): 0 119--139, 1997. URL: https://www.sciencedirect.com/science/article/pii/S002200009791504X

    1997

  12. [20]

    A Survey on Ensemble Learning for Data Stream Classification

    Heitor Murilo Gomes, Jean Paul Barddal, Fabrício Enembreck, and Albert Bifet. A Survey on Ensemble Learning for Data Stream Classification . ACM Computing Surveys, 50 0 (2): 0 1–36, March 2017. ISSN 1557-7341. URL: https://dl.acm.org/doi/10.1145/3054925

    2017 doi

  13. [21]

    The State of Sparse Training in Deep Reinforcement Learning

    Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro. The State of Sparse Training in Deep Reinforcement Learning . In International Conference on Machine Learning, pages 7766--7792. PMLR, 2022. URL: https://arxiv.org/abs/2206.10369

    2022 arXiv

  14. [22]

    Taylor, Mykola Pechenizkiy, and Decebal Constantin Mocanu

    Bram Grooten, Ghada Sokar, Shibhansh Dohare, Elena Mocanu, Matthew E. Taylor, Mykola Pechenizkiy, and Decebal Constantin Mocanu. Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learning . Int. Conf. Autonomous Agents and Multiagent Systems (AAMAS),...

    2023 arXiv

  15. [23]

    On Calibration of Modern Neural Networks

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On Calibration of Modern Neural Networks . In International Conference on Machine Learning, pages 1321--1330. PMLR, 2017. URL: https://proceedings.mlr.press/v70/guo17a.html

    2017

  16. [24]

    Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both Weights and Connections for Efficient Neural Networks . Advances in Neural Information Processing Systems, 2015. URL: https://arxiv.org/abs/1506.02626

    2015 arXiv

  17. [25]

    Neural Network Ensembles

    Lars Kai Hansen and Peter Salamon. Neural Network Ensembles . IEEE Transactions on Pattern Analysis and Machine Intelligence, 12 0 (10): 0 993--1001, 1990. URL: https://ieeexplore.ieee.org/document/58871

    1990

  18. [26]

    The Elements of Statistical Learning

    Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning . Springer Series in Statistics. Springer New York Inc., 2001. URL: https://link.springer.com/book/10.1007/978-0-387-84858-7

    2001 doi

  19. [27]

    Training independent subnetworks for robust prediction

    Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew Mingbo Dai, and Dustin Tran. Training independent subnetworks for robust prediction . In International Conference on Learning Representations, 2021. URL: https://a...

    2021 arXiv

  20. [28]

    Deep Residual Learning for Image Recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016. URL: https://arxiv.org/abs/1512.03385

    2016 arXiv

  21. [29]

    Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

    Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations . In International Conference on Learning Representations, 2019. URL: https://arxiv.org/abs/1903.12261

    2019 arXiv

  22. [30]

    Measuring Massive Multitask Language Understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding . International Conference for Learning Representations, 2021. URL: https://arxiv.org/abs/2009.03300

    2021 arXiv

  23. [31]

    Distilling the Knowledge in a Neural Network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network . Neural Information Processing Systems (Deep Learning Workshop), 2015. URL: https://arxiv.org/abs/1503.02531

    2015 arXiv

  24. [32]

    Jacobs, Michael I

    Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive Mixtures of Local Experts . Neural Computation, 3 0 (1): 0 79--87, 1991. URL: https://www.cs.toronto.edu/ fritz/absps/jjnh91.pdf

    1991

  25. [33]

    Joint Training of Deep Ensembles Fails Due to Learner Collusion

    Alan Jeffares, Tennison Liu, Jonathan Crabbé, and Mihaela van der Schaar. Joint Training of Deep Ensembles Fails Due to Learner Collusion . Advances in Neural Information Processing Systems, 2023. URL: https://arxiv.org/abs/2301.11323

    2023 arXiv

  26. [34]

    Mercer, Lalit R

    Frederick Jelinek, Robert L. Mercer, Lalit R. Bahl, and James K. Baker. Perplexity-a measure of the difficulty of speech recognition tasks . In Proceedings of the IEEE Symposium on Speech Recognition, pages 393--398, 1977. URL: https://pubs.aip.org/asa/jasa/article/62/S1/S63/6...

    1977

  27. [35]

    Adam: A Method for Stochastic Optimization

    Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization . International Conference for Learning Representations, 2015. URL: https://arxiv.org/abs/1412.6980

    2015 arXiv

  28. [36]

    Learning Multiple Layers of Features from Tiny Images

    Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images . Technical report, University of Toronto, 2009. URL: https://www.cs.toronto.edu/ kriz/learning-features-2009-TR.pdf

    2009

  29. [37]

    Kuncheva and Christopher J

    Ludmila I. Kuncheva and Christopher J. Whitaker. Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy . Kluwer Academic Publishers, 2003. URL: https://doi.org/10.1023/A:1022859003006

    2003 doi

  30. [38]

    Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles . Advances in Neural Information Processing Systems, 2017. URL: https://arxiv.org/abs/1612.01474

    2017 arXiv

  31. [39]

    Ya Le and Xuan S. Yang. Tiny ImageNet Visual Recognition Challenge , 2015. URL: https://cs231n.stanford.edu/reports/2015/pdfs/yle_project.pdf

    2015

  32. [40]

    Optimal Brain Damage

    Yann LeCun, John Denker, and Sara Solla. Optimal Brain Damage . In Advances in Neural Information Processing Systems, 1989. URL: https://proceedings.neurips.cc/paper_files/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf

    1989

  33. [41]

    Network Fission Ensembles for Low-Cost Self-Ensembles

    Hojung Lee and Jong-Seok Lee. Network Fission Ensembles for Low-Cost Self-Ensembles . 2024. URL: https://arxiv.org/abs/2408.02301

    2024 arXiv

  34. [42]

    Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. SNIP: Single-shot Network Pruning based on Connection Sensitivity . International Conference on Learning Representations, 2018. URL: https://arxiv.org/abs/1810.02340

    2018 arXiv

  35. [43]

    Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks , 2015

    Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks , 2015. URL: https://arxiv.org/abs/1511.06314

    2015 arXiv

  36. [44]

    Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN

    Pengxiang Li, Lu Yin, and Shiwei Liu. Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN . International Conference on Learning Representations, 2025. URL: https://arxiv.org/abs/2412.13795

    2025 arXiv

  37. [45]

    Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware

    Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, and Mykola Pechenizkiy. Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware. Neural Computing and Applications, 33: 0 2589--2604, 2021 a . URL: ...

    2021 doi

  38. [46]

    Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training

    Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training . International Conference on Machine Learning, 2021 b . URL: https://arxiv.org/abs/2102.02887

    2021 arXiv

  39. [47]

    Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity

    Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elena Mocanu, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity . International ...

    2022

  40. [48]

    The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training

    Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training . In Proceedings of the 11th International Conference on...

    2023 arXiv

  41. [49]

    Richard Maclin and David W. Opitz. Popular Ensemble Methods: An Empirical Study . Journal of Artificial Intelligence Research, 11: 0 169--198, 2011. URL: https://arxiv.org/abs/1106.0257

    2011 arXiv

  42. [50]

    Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , 2018

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , 2018. URL: https://arxiv.org/abs/1809.02789

    2018 arXiv

  43. [51]

    Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science

    Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science . Nature communications, 9 0 (1): 0 1--12, 2018. URL: htt...

    2018 arXiv

  44. [52]

    Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment

    Michael C Mozer and Paul Smolensky. Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment . In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 1. Morgan-Kaufmann, 1988. URL: https://proceedings.neurips.cc/pape...

    1988

  45. [53]

    Obtaining Well Calibrated Probabilities Using Bayesian Binning

    Mahdi Pakdaman Naeini, Gregory F Cooper, and Milos Hauskrecht. Obtaining Well Calibrated Probabilities Using Bayesian Binning . In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015. URL: https://doi.org/10.1609/aaai.v29i1.9602

    2015 doi

  46. [54]

    DeepSparse Inference Engine , 2021

    NeuralMagic. DeepSparse Inference Engine , 2021. URL: https://neuralmagic.com/deepsparse

    2021

  47. [55]

    Nowak, Bram Grooten, Decebal Constantin Mocanu, and Jacek Tabor

    Aleksandra I. Nowak, Bram Grooten, Decebal Constantin Mocanu, and Jacek Tabor. Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training . Advances in Neural Information Processing Systems, 36: 0 55160--55192, 2023. URL: https://arxiv.org/abs/2306.12230

    2023 arXiv

  48. [56]

    Nowak, ukasz Gniecki, Filip Szatkowski, and Jacek Tabor

    Aleksandra I. Nowak, ukasz Gniecki, Filip Szatkowski, and Jacek Tabor. Sparser, Better, Deeper, Stronger: Improving Sparse Training with Exact Orthogonal Initialization . In Proc. of the 41st International Conference on Machine Learning, 2024. URL: https://arxiv.org/abs/2406.01755

    2024 arXiv

  49. [57]

    ResNet50 v1.5 for PyTorch , 2024

    NVIDIA . ResNet50 v1.5 for PyTorch , 2024. URL: https://catalog.ngc.nvidia.com/orgs/nvidia/resources/resnet_50_v1_5_for_pytorch

    2024

  50. [58]

    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer . Journal of Machine Learning Research, 21 0 (140): 0 1--67, 2020. URL...

    2020

  51. [59]

    A Stochastic Approximation Method

    Herbert Robbins and Sutton Monro. A Stochastic Approximation Method . The Annals of Mathematical Statistics, 22 0 (3): 0 400--407, 1951. ISSN 00034851. URL: https://www.columbia.edu/ ww2040/8100F16/RM51.pdf

    1951

  52. [60]

    WinoGrande: An Adversarial Winograd Schema Challenge at Scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale . 2019. URL: http://arxiv.org/abs/1907.10641

    2019 arXiv

  53. [61]

    Towards Memory-Efficient Training for Extremely Large Output Spaces -- Learning with 500k Labels on a Single Commodity GPU

    Erik Schultheis and Rohit Babbar. Towards Memory-Efficient Training for Extremely Large Output Spaces -- Learning with 500k Labels on a Single Commodity GPU . In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 689--704. Springer, 2023....

    2023 arXiv

  54. [62]

    David B. Skalak. The Sources of Increased Accuracy for Two Proposed Boosting Algorithms . In AAAI Conference on Artificial Intelligence, 1996. URL: https://citeseerx.ist.psu.edu/document?doi=fa8115f79d8b951b71c964fc0401d5a716d516ec

    1996

  55. [63]

    Dynamic Sparse Training for Deep Reinforcement Learning

    Ghada Sokar, Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, and Peter Stone. Dynamic Sparse Training for Deep Reinforcement Learning . International Joint Conference on Artificial Intelligence, 2022. URL: https://arxiv.org/abs/2106.04217

    2022 arXiv

  56. [64]

    RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch

    Yiqin Tan, Pihe Hu, Ling Pan, Jiatai Huang, and Longbo Huang. RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch . In The Eleventh International Conference on Learning Representations, 2023. URL: https://arxiv.org/abs/2205.15043

    2023 arXiv

  57. [65]

    LLaMA: Open and Efficient Foundation Language Models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and Efficient Foundation Language Models . 2023. URL: https://arxiv.org/abs/2302.13971

    2023 arXiv

  58. [66]

    Varrette, H

    S. Varrette, H. Cartiaux, S. Peter, E. Kieffer, T. Valette, and A. Olloh. Management of an Academic HPC & Research Computing Facility: The ULHPC Experience 2.0 . In ACM High Performance Computing and Cluster Technologies, 2022. URL: https://doi.org/10.1145/3560442.3560445

    2022

  59. [67]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need . Advances in Neural Information Processing Systems, 2017. URL: https://arxiv.org/abs/1706.03762

    2017 arXiv

  60. [68]

    Picking Winning Tickets Before Training by Preserving Gradient Flow

    Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking Winning Tickets Before Training by Preserving Gradient Flow . International Conference on Learning Representations, 2020. URL: https://arxiv.org/abs/2002.07376

    2020 arXiv

  61. [69]

    Learning Robust Global Representations by Penalizing Local Predictive Power

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning Robust Global Representations by Penalizing Local Predictive Power . In Neural Information Processing Systems, 2019. URL: https://papers.nips.cc/paper_files/paper/2019/file/3eefceb8087e964f89c2d59e8a249915-Paper.pdf

    2019

  62. [70]

    BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning

    Yeming Wen, Dustin Tran, and Jimmy Ba. BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning . International Conference on Learning Representations, 2020. URL: https://arxiv.org/abs/2002.06715

    2020 arXiv

  63. [71]

    Nerva: a Truly Sparse Implementation of Neural Networks

    Wieger Wesselink, Bram Grooten, Qiao Xiao, Cassio de Campos, and Mykola Pechenizkiy. Nerva: a Truly Sparse Implementation of Neural Networks . arXiv preprint arXiv:2407.17437, 2024. URL: https://arxiv.org/abs/2407.17437

    2024 arXiv

  64. [72]

    Prune and Tune Ensembles: Low-Cost Ensemble Learning with Sparse Independent Subnetworks

    Tim Whitaker and Darrell Whitley. Prune and Tune Ensembles: Low-Cost Ensemble Learning with Sparse Independent Subnetworks . In AAAI, pages 8638--8646. AAAI Press, 2022. ISBN 978-1-57735-876-3. URL: https://arxiv.org/abs/2202.11782

    2022 arXiv

  65. [73]

    HuggingFace's Transformers: State-of-the-art Natural Language Processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. HuggingFace's Transformers: State-of-the-art Natural Language Processing . 2019. URL: https://arxiv.org/abs/1910.03771

    2019 arXiv

  66. [74]

    David H. Wolpert. Stacked generalization . Neural Networks, 5 0 (2): 0 241--259, 1992. URL: https://www.sciencedirect.com/science/article/abs/pii/S0893608005800231

    1992

  67. [75]

    Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness

    Boqian Wu, Qiao Xiao, Shunxin Wang, Nicola Strisciuglio, Mykola Pechenizkiy, Maurice van Keulen, Decebal Constantin Mocanu, and Elena Mocanu. Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness . International Conference on Learn...

    2025 arXiv

  68. [76]

    Continual Learning with Dynamic Sparse Training: Exploring Algorithms for Effective Model Updates

    Murat Onur Yildirim, Elif Ceren Gok Yildirim, Ghada Sokar, Decebal Constantin Mocanu, and Joaquin Vanschoren. Continual Learning with Dynamic Sparse Training: Exploring Algorithms for Effective Model Updates . Conference on Parsimony and Learning, 2024. URL: https://arxiv.org/...

    2024 arXiv

  69. [77]

    Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity

    Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, et al. Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity . International Conference on Machine Learning, ...

    2023 arXiv

  70. [78]

    Baraniuk, Zhangyang Wang, and Yingyan Lin

    Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks . International Conference on Learning Representations, 2020. URL: https:...

    2020 arXiv

  71. [79]

    MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge

    Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, Siyue Wang, Minghai Qin, Bin Ren, Yanzhi Wang, Sijia Liu, and Xue Lin. MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge . In Advan...

    2021 arXiv

  72. [80]

    Wide Residual Networks

    Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks . CoRR, abs/1605.07146, 2016. URL: https://arxiv.org/abs/1605.07146

    2016 arXiv

  73. [81]

    HellaSwag: Can a Machine Really Finish Your Sentence? Association for Computational Linguistics, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence? Association for Computational Linguistics, 2019. URL: https://arxiv.org/abs/1905.07830

    2019 arXiv

  74. [82]

    Brain-inspired sparse training enables Transformers and LLMs to perform as fully connected

    Yingtao Zhang, Jialin Zhao, Wenjing Wu, Ziheng Liao, Umberto Michieli, and Carlo Vittorio Cannistraci. Brain-inspired sparse training enables Transformers and LLMs to perform as fully connected . arXiv preprint arXiv:2501.19107, 2025. URL: https://arxiv.org/abs/2501.19107

    2025

  75. [83]

    GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection . International Conference on Machine Learning, 2024. URL: https://arxiv.org/abs/2403.03507

    2024 arXiv

  76. [84]

    Robust Lottery Tickets for Pre-trained Language Models

    Rui Zheng, Bao Rong, Yuhao Zhou, Di Liang, Sirui Wang, Wei Wu, Tao Gui, Qi Zhang, and Xuanjing Huang. Robust Lottery Tickets for Pre-trained Language Models . In Association for Computational Linguistics, page 2211–2224, 2022. URL: https://arxiv.org/abs/2211.03013

    2022 arXiv

  77. [85]

    Ensemble Methods: Foundations and Algorithms

    Zhi-Hua Zhou. Ensemble Methods: Foundations and Algorithms . Chapman & Hall/CRC press, 2012. URL: https://dl.acm.org/doi/10.5555/2381019

    2012 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.