Pith. sign in

REVIEW 9 cited by

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2004.02984 v2 pith:LKHMTHKZ submitted 2020-04-06 cs.CL cs.LG

MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices

Zhiqing Sun , Hongkun Yu , Xiaodan Song , Renjie Liu , Yiming Yang , Denny Zhou This is my paper
classification cs.CL cs.LG
keywords mobilebertbertmodelbaseachievesdesigneddeviceslanguage
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds of millions of parameters. However, these models suffer from heavy model sizes and high latency such that they cannot be deployed to resource-limited mobile devices. In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model. Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to various downstream NLP tasks via simple fine-tuning. Basically, MobileBERT is a thin version of BERT_LARGE, while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks. To train MobileBERT, we first train a specially designed teacher model, an inverted-bottleneck incorporated BERT_LARGE model. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical studies show that MobileBERT is 4.3x smaller and 5.5x faster than BERT_BASE while achieving competitive results on well-known benchmarks. On the natural language inference tasks of GLUE, MobileBERT achieves a GLUEscore o 77.7 (0.6 lower than BERT_BASE), and 62 ms latency on a Pixel 4 phone. On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a dev F1 score of 90.0/79.2 (1.5/2.1 higher than BERT_BASE).

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

    cs.CL 2023-05 conditional novelty 8.0

    Tiny language models under 10M parameters trained on a synthetic children's story dataset generate fluent, consistent, multi-paragraph English text with near-perfect grammar and reasoning.

  2. Faster or Stronger: Towards Flexible Visual Place Recognition via Weighted Aggregation and Token Pruning

    cs.CV 2026-05 unverdicted novelty 7.0

    Proposes weighted aggregation of clusters and self-distillation-driven token pruning to improve both accuracy and efficiency in ViT-based visual place recognition.

  3. Depth Adaptive Efficient Visual Autoregressive Modeling

    cs.CV 2026-04 unverdicted novelty 7.0

    DepthVAR adaptively allocates per-token computational depth in VAR models using a cyclic rotated scheduler and dynamic layer masking to achieve 2.3-3.1x inference speedup with minimal quality loss.

  4. SCOUT: A Defense Against Data Poisoning Attacks in Fine-Tuned Language Models

    cs.CR 2025-12 unverdicted novelty 6.0

    SCOUT uses token saliency analysis to detect both standard and contextually-plausible backdoor attacks in language models while maintaining clean accuracy.

  5. LEAF: Knowledge Distillation of Text Embedding Models with Teacher-Aligned Representations

    cs.IR 2025-09 conditional novelty 6.0

    LEAF distills teacher-aligned student embedding models that achieve new SOTA results on BEIR and MTEB for their size class while requiring only modest data and compute.

  6. ESsEN: Training Compact Discriminative Vision-Language Transformers in a Low-Resource Setting

    cs.CV 2026-04 unverdicted novelty 5.0

    ESsEN is a parameter-efficient two-tower vision-language transformer that matches larger models on discriminative tasks after training end-to-end with limited data and resources.

  7. Fast Transformer Inference on ARM-Based HMPSoCs

    cs.AR 2026-06 unverdicted novelty 4.0

    Extends ARM-CL with transformer kernels and CPU-GPU cooperative inference to achieve up to 3x faster inference and 15.72% additional latency reduction on ARM HMPSoCs.

  8. Little Brains, Big Feats: Exploring Compact Language Models

    cs.CL 2026-06 unverdicted novelty 3.0

    Small language models can run RAG generation on-device without GPUs in reasonable time.

  9. A Survey on Foundation Models for Personalized Federated Intelligence

    cs.AI 2025-05 unverdicted novelty 3.0

    The survey introduces personalized federated intelligence (PFI) as a framework integrating federated learning and foundation models to support privacy-aware personalization of AI models.