Pith. sign in

REVIEW 9 cited by

Large scale distributed neural network training through online distillation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1804.03235 v2 pith:SZ32ZXJZ submitted 2018-04-09 cs.LG cs.AIstat.ML

Rohan Anil , Gabriel Pereyra , Alexandre Passos , Robert Ormandi , George E. Dahl , Geoffrey E. Hinton This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords distillationmodelneuralonlinepredictionstrainingclaimdata
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

Techniques such as ensembling and distillation promise model quality improvements when paired with almost any base model. However, due to increased test-time cost (for ensembles) and increased complexity of the training pipeline (for distillation), these techniques are challenging to use in industrial settings. In this paper we explore a variant of distillation which is relatively straightforward to use as it does not require a complicated multi-stage setup or many new hyperparameters. Our first claim is that online distillation enables us to use extra parallelism to fit very large datasets about twice as fast. Crucially, we can still speed up training even after we have already reached the point at which additional parallelism provides no benefit for synchronous or asynchronous stochastic gradient descent. Two neural networks trained on disjoint subsets of the data can share knowledge by encouraging each model to agree with the predictions the other model would have made. These predictions can come from a stale version of the other model so they can be safely computed using weights that only rarely get transmitted. Our second claim is that online distillation is a cost-effective way to make the exact predictions of a model dramatically more reproducible. We support our claims using experiments on the Criteo Display Ad Challenge dataset, ImageNet, and the largest to-date dataset used for neural language modeling, containing $6\times 10^{11}$ tokens and based on the Common Crawl repository of web data.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 152 citations worldwide. Full citation record

  1. Bridging Compute- and Data-Optimal Pretraining

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.

  2. Slot2Text: Object-Centric Visual Tokenization for Efficient and Spatially Traceable Surgical MLLMs

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Slot2Text shows that nine region-bearing slot tokens, not hundreds of patch tokens, can support competitive surgical VQA and spatial grounding.

  3. Optimal Transceiver Design in Over-the-Air Federated Distillation

    eess.SP 2025-07 reject novelty 6.0 of 10

    An over-the-air federated distillation scheme with closed-form power control and SDR-based beamforming is derived from a convergence-rate bound, with a claimed proof that the beamforming relaxation is tight.

  4. Out-of-Distribution Federated Distillation with Domain-Aware Proxy

    cs.LG 2026-08 reject novelty 4.0 of 10

    DPS-FD, a domain-aware proxy selection method with a vocabulary-constrained LLM proxy generator, is claimed to improve out-of-distribution federated distillation, but its OOD evaluation is compromised by proxy data dr...

  5. Beyond Model Scale Limits: End-Edge-Cloud Federated Learning with Self-Rectified Knowledge Agglomeration

    cs.DC 2025-01 conditional novelty 4.0 of 10

    FedEEC trains larger cloud models in end-edge-cloud federated learning via bridge-sample distillation and self-rectified knowledge transfer, reporting accuracy gains on SVHN, CIFAR-10, and CINIC-10.

  6. A Collaborative Ensemble Framework for CTR Prediction

    cs.IR 2024-11 conditional novelty 4.0 of 10

    CETNet combines two CTR models with separate embeddings, KL collaboration, and entropy-based confidence fusion, yielding small AUC gains on public benchmarks but not in the internal deployment.

  7. Modular Federated Learning: A Meta-Framework Perspective

    cs.LG 2025-05 conditional novelty 3.0 of 10

    A 63-page survey that reframes federated learning as a composition of eight modules and proposes an 'alignment operator' taxonomy, while surveying Python FL frameworks and open challenges.

  8. Learning Representations and Agents for Information Retrieval

    cs.IR 2019-08 conditional novelty 3.0 of 10

    A dissertation showing that a BERT re-ranker combined with document expansion by predicted queries roughly doubles BM25 retrieval effectiveness on MS MARCO and TREC-CAR.

  9. Distilling On-Device Intelligence at the Network Edge

    cs.IT 2019-08 unverdicted novelty 3.0 of 10

    A review article that organizes communication-efficient and privacy-preserving on-device federated learning methods into three exchange modes and illustrates seven example frameworks.

Pith tools