Pith. sign in

REVIEW 5 cited by

FastBERT: a Self-distilling BERT with Adaptive Inference Time

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2004.02178 v2 pith:GJWVK4GO submitted 2020-04-05 cs.CL

Weijie Liu , Peng Zhou , Zhe Zhao , Zhiruo Wang , Haotang Deng , Qi Ju This is my paper
classification cs.CL
keywords bertinferencemodeladaptivefastbertmodelsperformancespeed
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Pre-trained language models like BERT have proven to be highly performant. However, they are often computationally expensive in many practical scenarios, for such heavy models can hardly be readily implemented with limited resources. To improve their efficiency with an assured model performance, we propose a novel speed-tunable FastBERT with adaptive inference time. The speed at inference can be flexibly adjusted under varying demands, while redundant calculation of samples is avoided. Moreover, this model adopts a unique self-distillation mechanism at fine-tuning, further enabling a greater computational efficacy with minimal loss in performance. Our model achieves promising results in twelve English and Chinese datasets. It is able to speed up by a wide range from 1 to 12 times than BERT if given different speedup thresholds to make a speed-performance tradeoff.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GNN-CNN: An Efficient Hybrid Model of Convolutional and Graph Neural Networks for Text Representation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A hybrid GNN-CNN text classifier with real-time graph generation and injected LLM embeddings achieves near-transformer accuracy at linear complexity.

  2. FREE: Fast and Robust Vision Language Models with Early Exits

    cs.LG 2025-06 conditional novelty 6.0 of 10

    An adversarial early-exit method for frozen-backbone vision language models that reuses the final classifier and reports 1.5x inference speedup with comparable accuracy.

  3. Harnessing Input-Adaptive Inference for Efficient VLN

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A three-part input-adaptive inference method (view masking, adaptive early exit, view caching) cuts VLN computation roughly in half on seven benchmarks with moderate success-rate loss.

  4. Resource-Efficient Automatic Software Vulnerability Assessment via Knowledge Distillation and Particle Swarm Optimization

    cs.LG 2025-07 conditional novelty 4.0 of 10

    A PSO-guided knowledge distillation framework compresses a CodeBERT vulnerability assessor to 0.6% of its original size while retaining 89.3% of its accuracy.

  5. Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs

    cs.LG 2025-07 reject novelty 4.0 of 10

    Pretrained LLM layers can be skipped/repeated per input to build custom paths, but the search uses ground-truth answers, so the accuracy gains are fitted, not predicted.

Pith tools