Pith. sign in

REVIEW 56 cited by

TinyBERT: Distilling BERT for Natural Language Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1909.10351 v5 pith:NCY5SNFA submitted 2019-09-23 cs.CL cs.AIcs.LG

Xiaoqi Jiao , Yichun Yin , Lifeng Shang , Xin Jiang , Xiao Chen , Linlin Li , Fang Wang , Qun Liu This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords tinybertbertdistillationlanguageinferenceknowledgelayersteacher
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually computationally expensive, so it is difficult to efficiently execute them on resource-restricted devices. To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models. By leveraging this new KD method, the plenty of knowledge encoded in a large teacher BERT can be effectively transferred to a small student Tiny-BERT. Then, we introduce a new two-stage learning framework for TinyBERT, which performs Transformer distillation at both the pretraining and task-specific learning stages. This framework ensures that TinyBERT can capture he general-domain as well as the task-specific knowledge in BERT. TinyBERT with 4 layers is empirically effective and achieves more than 96.8% the performance of its teacher BERTBASE on GLUE benchmark, while being 7.5x smaller and 9.4x faster on inference. TinyBERT with 4 layers is also significantly better than 4-layer state-of-the-art baselines on BERT distillation, with only about 28% parameters and about 31% inference time of them. Moreover, TinyBERT with 6 layers performs on-par with its teacher BERTBASE.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 56 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 137 citations worldwide. Full citation record

  1. On Teacher Hacking in Language Model Distillation

    cs.LG 2025-02 conditional novelty 7.0 of 10

    Distilling a student from an imperfect teacher on a fixed offline dataset can make the student match the teacher's errors while drifting from the true target, a phenomenon the authors call teacher hacking.

  2. Active Data Curation Effectively Distills Large-Scale Multimodal Models

    cs.CV 2024-11 conditional novelty 7.0 of 10

    Selecting training data by a reference model's loss acts as an implicit distillation, and combining it with explicit distillation yields more FLOP-efficient vision-language models that beat prior SoTA on 27 benchmarks.

  3. DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce

    cs.LG 2026-02 conditional novelty 6.0 of 10

    DynamiQ compresses gradients for multi-hop all-reduce with per-block bit-width allocation, block reordering, and fused recompression, matching BF16 accuracy at ~5 bits per coordinate and improving time-to-accuracy by ...

  4. Uncovering Gradient Inversion Risks in Practical Language Model Training

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A hybrid continuous-discrete optimization attack, Grab, recovers up to 92.9% of private training text from gradients in federated learning of masked language models under practical settings.

  5. Multimodal Medical Image Binding via Shared Text Embeddings

    eess.IV 2025-06 conditional novelty 6.0 of 10

    Five modality-specific CLIP-like medical models are aligned through a shared, distilled text embedding space, enabling zero-shot cross-modal retrieval and improved few-shot classification without paired image data.

  6. Knowledge Distillation Framework for Accelerating High-Accuracy Neural Network-Based Molecular Dynamics Simulations

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A knowledge-distillation framework that uses an off-the-shelf pre-trained neural network potential as teacher, followed by a small density-functional-theory fine-tuning set, produces fast and accurate molecular dynami...

  7. Decoupled Global-Local Alignment for Improving Compositional Understanding

    cs.CV 2025-04 conditional novelty 6.0 of 10

    DeGLA fine-tunes CLIP with LLM-generated hard negatives plus EMA self-distillation, improving compositional reasoning benchmarks by 1.9 to 4.9 points over CE-CLIP while staying within 2.3 points of the original CLIP o...

  8. Optimizing Knowledge Distillation in Transformers: Enabling Multi-Head Attention without Alignment Barriers

    cs.CV 2025-02 reject novelty 6.0 of 10

    Squeezing-Heads Distillation mixes several teacher attention maps into one per-sample weighted map, enabling knowledge distillation between transformers with different head counts without extra parameters.

  9. Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile

    cs.CV 2025-02 conditional novelty 6.0 of 10

    A three-stage pipeline combining sparse 'tile' attention with multi-step consistency distillation makes Open-Sora-Plan video generation up to 7.8x faster while keeping the aggregate VBench final score within 1%.

  10. BEEM: Boosting Performance of Early Exit DNNs using Multi-Exit Classifiers as Experts

    cs.LG 2025-02 conditional novelty 6.0 of 10

    BEEM aggregates weighted confidence from consistent neighboring exit classifiers, resetting on disagreement, and sets thresholds from validation error rates to accelerate early-exit inference.

  11. NewsEdits 2.0: Learning the Intentions Behind Updating News

    cs.CL 2024-11 conditional novelty 6.0 of 10

    NewsEdits 2.0 introduces an edit-intention taxonomy and text-based models that predict factual updates in news revisions, enabling LLMs to abstain from answering with outdated facts at near-oracle accuracy.

  12. FLARE: FP-Less PTQ and Low-ENOB ADC Based AMS-PiM for Error-Resilient, Fast, and Efficient Transformer Acceleration

    cs.LG 2024-11 conditional novelty 6.0 of 10

    FLARE replaces floating-point quantization and softmax in transformers with integer-only eMSB-based methods and low-ENOB analog-to-digital converters, aiming for fast and low-energy attention.

  13. FHECore: Rethinking GPU Microarchitecture for Fully Homomorphic Encryption

    cs.AR 2026-02 reject novelty 5.0 of 10

    A GPU-integrated 16×8 systolic modulo-multiply-accumulate unit with Barrett reduction is claimed to speed CKKS NTT and base conversion by roughly 1.6–2.1x at 2.4% area cost.

  14. SindBERT, the Sailor: Charting the Seas of Turkish NLP

    cs.CL 2025-10 conditional novelty 5.0 of 10

    SindBERT releases Turkish RoBERTa base/large models trained on 312GB of text; they match existing models, with the large variant best on two of four tasks and little scaling gain.

  15. Expandable Residual Approximation for Knowledge Distillation

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A new knowledge distillation method decomposes the teacher-student feature gap into multiple residual steps and reports improved accuracy on ImageNet and COCO.

  16. Checkmate: interpretable and explainable RSVQA is the endgame

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Checkmate answers RSVQA questions by reading a cell-level text summary of a segmentation map, and the new Chessboard dataset reduces answer bias, yet the cell grounding metric is partly solvable from question text alone.

  17. GeRe: Towards Efficient Anti-Forgetting in Continual Learning of LLM via General Samples Replay

    cs.CL 2025-08 conditional novelty 5.0 of 10

    A fixed 1,000-sample replay set from pretraining text, trained with a threshold-based margin loss on last-layer hidden states, reduces forgetting across 15 sequential finetuning tasks in Llama-3.1-8B.

  18. Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Structural pruning with finetuning plus hidden-state distillation recovers most performance in multimodal LLMs, with 5% of training data sufficient at moderate compression levels.

  19. AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes

    cs.AI 2025-06 reject novelty 5.0 of 10

    AgentDistill distills agent capabilities without any training by having a teacher generate reusable MCP tool boxes that small-model students invoke at inference time.

  20. Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Data

    cs.LG 2025-05 conditional novelty 5.0 of 10

    FedProj combines client-side gradient projection onto a global-knowledge loss with server-side ensemble distillation and outperforms existing federated learning methods on non-IID image and NLP benchmarks.

  21. A Survey of LLM $\times$ DATA

    cs.DB 2025-05 conditional novelty 5.0 of 10

    A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.

  22. Accelerating Clinical NLP at Scale with a Hybrid Framework with Reduced GPU Demands: A Case Study in Dementia Identification

    cs.CL 2025-04 conditional novelty 5.0 of 10

    A rule-SVM-BERT hybrid identified dementia from 2.1 billion clinical notes with patient-level F1 of 0.87, roughly two weeks of GPU time, and three times the case yield of structured codes.

  23. Fake News Detection After LLM Laundering: Measurement and Explanation

    cs.CL 2025-01 conditional novelty 5.0 of 10

    LLM paraphrasing of fake news degrades detector performance across 17 detectors, with Pegasus evading best and a sentiment shift that BERTScore fails to capture.

  24. Evaluating the Effectiveness of XAI Techniques for Encoder-Based Language Models

    cs.CL 2025-01 conditional novelty 5.0 of 10

    A four-metric benchmark of six XAI methods on encoder language models finds LIME best on human agreement, AMV best on robustness and consistency, and LRP best on contrastivity.

  25. Extracting General-use Transformers for Low-resource Languages via Knowledge Distillation

    cs.CL 2025-01 conditional novelty 5.0 of 10

    Simple knowledge distillation from mBERT produces smaller, faster Tagalog-only transformers that match the teacher on some tasks and lag on NER.

  26. Lillama: Large Language Models Compression via Low-Rank Feature Distillation

    cs.LG 2024-12 conditional novelty 5.0 of 10

    Lillama compresses LLMs by SVD-initialized low-rank layers trained with a local Teacher plus Student activation distillation loss, achieving 20-40% parameter reduction with only 13 million calibration tokens.

  27. Distributed Inference on Mobile Edge and Cloud: A Data-Cartography based Clustering Approach

    cs.DC 2024-12 conditional novelty 5.0 of 10

    DIMEC-DC uses data-cartography clusters of validation embeddings to assign each input to mobile, edge, or cloud inference, cutting cost by over 43 percent with near-cloud accuracy on GLUE tasks.

  28. Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Models

    cs.CL 2024-11 conditional novelty 5.0 of 10

    DynSDPB fine-tunes small language models by self-distilling soft labels from the previous mini-batch, with dynamic per-sample temperature and loss weighting.

  29. ElastiFormer: Learned Redundancy Reduction in Transformer via Self-Distillation

    cs.LG 2024-11 conditional novelty 5.0 of 10

    A post-training routing method that uses self-distillation to let frozen pretrained Transformers process only a subset of parameters and tokens, cutting active compute by 20 to 50 percent.

  30. NoteBar: An AI-Assisted Note-Taking System for Personal Knowledge Management

    cs.CL 2025-09 reject novelty 4.0 of 10

    NoteBar introduces a persona-conditioned synthetic note dataset and shows DeBERTa-v3 reaches 0.78 accuracy and 0.76 F1 on multi-label note classification, while claiming user-facing benefits it does not actually measure.

  31. Enhanced Arabic Text Retrieval with Attentive Relevance Scoring

    cs.CL 2025-07 conditional novelty 4.0 of 10

    An Arabic dense retriever using a trainable attentive scoring module instead of dot-product similarity reports improved top-k passage retrieval on ArabicaQA.

  32. Resource-Efficient Automatic Software Vulnerability Assessment via Knowledge Distillation and Particle Swarm Optimization

    cs.LG 2025-07 conditional novelty 4.0 of 10

    A PSO-guided knowledge distillation framework compresses a CodeBERT vulnerability assessor to 0.6% of its original size while retaining 89.3% of its accuracy.

  33. MobileRAG: A Fast, Memory-Efficient, and Energy-Efficient Method for On-Device RAG

    cs.DB 2025-07 conditional novelty 4.0 of 10

    A fully on-device RAG pipeline using a partitioned, partially disk-loaded graph index and selective sentence-window reduction claims 1.72-8.89x faster vector search and up to 40.2% lower power than baselines, with com...

  34. Toward Edge General Intelligence with Multiple-Large Language Model (Multi-LLM): Architecture, Trust, and Orchestration

    cs.NI 2025-07 conditional novelty 4.0 of 10

    A survey of multi-LLM systems in edge computing, covering architectures, enabling technologies, trust mechanisms, applications, and open datasets for edge general intelligence.

  35. SEMFED: Semantic-Aware Resource-Efficient Federated Learning for Heterogeneous NLP Tasks

    cs.CL 2025-05 reject novelty 4.0 of 10

    A framework combining semantic-aware client selection, heterogeneous model sizes, and feature compression reports 98.5% accuracy and 80.5% less communication on a synthetic 10-client NLP task.

  36. Private Transformer Inference in MLaaS: A Survey

    cs.CR 2025-05 conditional novelty 4.0 of 10

    A structured survey of private transformer inference, comparing MPC- and HE-based methods and showing non-linear layers dominate overhead.

  37. LLM Modules: Knowledge Transfer from a Large to a Small Model using Enhanced Cross-Attention

    cs.CL 2025-02 reject novelty 4.0 of 10

    A frozen Qwen2-1.5B teacher passes its hidden states through gated cross-attention into GPT-Neo-125M, which after 15 epochs generates more coherent arithmetic responses than the base small model.

  38. Compact Bayesian Neural Networks via pruned MCMC sampling

    cs.LG 2025-01 conditional novelty 4.0 of 10

    Post-hoc signal-to-noise and signal-plus-noise pruning with a short resampling run can shrink MCMC-trained Bayesian neural networks by 75% with modest accuracy loss, though uncertainty retention is not measured.

  39. YuLan-Mini: An Open Data-efficient Language Model

    cs.CL 2024-12 conditional novelty 4.0 of 10

    A 2.42B-parameter base model trained on 1.08T tokens matches or beats several industry baselines trained on 7T to 18T tokens across math, code, and general benchmarks.

  40. Deploying Foundation Model Powered Agent Services: A Survey

    cs.DC 2024-12 accept novelty 4.0 of 10

    This survey proposes a layered framework (execution, resource, model, agent, application) for deploying foundation-model-powered agent services across edge-cloud environments, and reviews optimization techniques at ea...

  41. The Generative AI Ethics Playbook

    cs.CY 2024-12 conditional novelty 4.0 of 10

    A structured playbook that collects existing guidance, checklists, and case studies to help generative AI practitioners identify and mitigate ethical harms across six lifecycle stages.

  42. Enhancing CLIP Conceptual Embedding through Knowledge Distillation

    cs.AI 2024-12 reject novelty 4.0 of 10

    Knowledge-CLIP distills Llama 2 embeddings into CLIP and uses k-means soft concept labels to slightly improve CLIP text and image encoder scores on three benchmarks.

  43. Vision Mamba Distillation for Low-resolution Fine-grained Image Classification

    cs.CV 2024-11 conditional novelty 4.0 of 10

    A super-resolution-plus-Vision-Mamba student network distilled from a high-resolution teacher achieves new state-of-the-art accuracy on low-resolution fine-grained image classification with fewer parameters and FLOPs.

  44. SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference

    cs.AI 2026-08 conditional novelty 3.0 of 10

    SPADE places the draft model of speculative decoding on the edge and the verifier on the cloud, cutting cloud model calls by about 76% while preserving close to full-model quality.

  45. AMMKD: Adaptive Multimodal Multi-teacher Distillation for Lightweight Vision-Language Models

    cs.CV 2025-08 reject novelty 3.0 of 10

    AMMKD claims large gains from adaptively weighted two-teacher CLIP distillation, but its equations are internally inconsistent, its baselines are unverifiable, and its tests do not match its stated retrieval goal.

  46. KDH-MLTC: Knowledge Distillation for Healthcare Multi-Label Text Classification

    cs.CL 2025-05 conditional novelty 3.0 of 10

    A distilled BERT-to-DistilBERT multi-label classifier using sequential per-topic fine-tuning beats TF-IDF plus SVM and GPT-4o zero-shot on Hallmarks of Cancer abstracts when trained on 1,000 documents.

  47. LLM-Based Threat Detection and Prevention Framework for IoT Ecosystems

    cs.CR 2025-05 reject novelty 3.0 of 10

    Fine-tuned BERT-small reaches 99.75% test accuracy on combined IoT-23 and TON IoT attack classification, but the claimed real-time prevention advantage over traditional methods is not demonstrated.

  48. PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models

    cs.CL 2025-04 conditional novelty 3.0 of 10

    A survey that organizes PEFT methods into additive, selective, reparameterized, hybrid, and unified families, but with no new method or verified experiments.

  49. A Survey of Early Exit Deep Neural Networks in NLP

    cs.LG 2025-01 conditional novelty 3.0 of 10

    A review of early exit deep neural network methods in NLP that has no new experiments but organizes the existing literature.

  50. Text Classification: Neural Networks VS Machine Learning Models VS Pre-trained Models

    cs.LG 2024-12 conditional novelty 3.0 of 10

    Pretrained transformers, especially BERT, outperform standard neural networks and machine learning models on a two-level news classification benchmark, though ALBERT collapses at the finer level.

  51. Collaborative Distillation Strategies for Parameter-Efficient Language Model Deployment

    cs.CL 2025-07 reject novelty 2.0 of 10

    The paper claims that combining weighted output fusion, entropy-driven teacher weighting, and feature alignment produces a better distilled language model.

  52. Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptation

    cs.CL 2025-05 unverdicted novelty 2.0 of 10

    A survey of small language models that organizes known methods into taxonomies but adds no new models, data, or validated benchmarks.

  53. Performance Analysis of Traditional VQA Models Under Limited Computational Resources

    cs.CV 2025-02 reject novelty 2.0 of 10

    An empirical comparison claims BidGRU with embedding size 300 and vocabulary 3000 is the best resource-constrained VQA configuration, but the paper lacks dataset and statistical details.

  54. Can bidirectional encoder become the ultimate winner for downstream applications of foundation models?

    cs.CL 2024-11 unverdicted novelty 1.0 of 10

    A review of bidirectional encoder models (BERT and variants) and their performance on GLUE and SQuAD relative to one-way generative models.

  55. A Survey on Private Transformer Inference

    cs.CR 2024-12 reject

    A literature survey on private transformer inference that is too incomplete to support its promised comparisons and evaluation guidelines.

  56. Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

    cs.LG 2024-12 conditional

    A broad but error-prone survey of LLM and MLLM architectures, training methods, benchmarks, and challenges.

Pith tools