Pith. sign in

REVIEW 9 cited by

MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2004.02984 v2 pith:LKHMTHKZ submitted 2020-04-06 cs.CL cs.LG

Zhiqing Sun , Hongkun Yu , Xiaodan Song , Renjie Liu , Yiming Yang , Denny Zhou This is my paper
classification cs.CLcs.LG
keywords mobilebertbertmodelbaseachievesdesigneddeviceslanguage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds of millions of parameters. However, these models suffer from heavy model sizes and high latency such that they cannot be deployed to resource-limited mobile devices. In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model. Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to various downstream NLP tasks via simple fine-tuning. Basically, MobileBERT is a thin version of BERT_LARGE, while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks. To train MobileBERT, we first train a specially designed teacher model, an inverted-bottleneck incorporated BERT_LARGE model. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical studies show that MobileBERT is 4.3x smaller and 5.5x faster than BERT_BASE while achieving competitive results on well-known benchmarks. On the natural language inference tasks of GLUE, MobileBERT achieves a GLUEscore o 77.7 (0.6 lower than BERT_BASE), and 62 ms latency on a Pixel 4 phone. On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a dev F1 score of 90.0/79.2 (1.5/2.1 higher than BERT_BASE).

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SALAAD: Sparse And Low-Rank Adaptation via ADMM for Large Language Model Inference

    cs.LG 2026-02 conditional novelty 6.0 of 10

    SALAAD uses an ADMM-style optimizer and an adaptive controller to force LLM weights into sparse-plus-low-rank form during pretraining, so one checkpoint can be elastically deployed at many parameter budgets.

  2. FedAPM: Federated Learning via ADMM with Partial Model Personalization

    cs.LG 2025-06 conditional novelty 6.0 of 10

    FedAPM applies ADMM with first- and second-order proximal corrections to partial model personalization in federated learning, proving global convergence and reporting better accuracy, F1, and AUC than FedAlt, FedSim, ...

  3. Linearity-based neural network compression

    cs.LG 2025-06 conditional novelty 5.0 of 10

    A compression method removes near-always-active ReLU neurons and replaces them with shortcut connections, shrinking some fully connected models to one quarter of their size with modest accuracy loss.

  4. AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes

    cs.AI 2025-06 reject novelty 5.0 of 10

    AgentDistill distills agent capabilities without any training by having a teacher generate reusable MCP tool boxes that small-model students invoke at inference time.

  5. Interfaze: The Future of AI is built on Task-Specific Small Models

    cs.AI 2026-02 reject novelty 4.0 of 10

    Interfaze-Beta uses small specialist models and tools to build a compact context that a general-purpose LLM answers from, reporting competitive benchmark scores without reproducible evidence.

  6. MobileRAG: A Fast, Memory-Efficient, and Energy-Efficient Method for On-Device RAG

    cs.DB 2025-07 conditional novelty 4.0 of 10

    A fully on-device RAG pipeline using a partitioned, partially disk-loaded graph index and selective sentence-window reduction claims 1.72-8.89x faster vector search and up to 40.2% lower power than baselines, with com...

  7. ReasonBridge: Efficient Reasoning Transfer from Closed to Open-Source Language Models

    cs.AI 2025-06 conditional novelty 4.0 of 10

    ReasonBridge distills 1,000 curated reasoning traces from a closed model into open-source models via hierarchical adapters and guided inference, claiming up to 23% benchmark gains.

  8. Collaborative Distillation Strategies for Parameter-Efficient Language Model Deployment

    cs.CL 2025-07 reject novelty 2.0 of 10

    The paper claims that combining weighted output fusion, entropy-driven teacher weighting, and feature alignment produces a better distilled language model.

  9. Improving QA Efficiency with DistilBERT: Fine-Tuning and Inference on mobile Intel CPUs

    cs.CL 2025-05 conditional novelty 2.0 of 10

    Fine-tuned DistilBERT on 45,000 SQuAD examples achieves validation F1 0.6536 and 0.1208 s average inference per question on an Intel i7-1355U CPU.

Pith tools