Pith. sign in

REVIEW 3 major objections 5 minor 22 cited by

Patient Knowledge Distillation for BERT Model Compression

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Compressing BERT by distilling knowledge from several of the teacher's intermediate layers, not just its final output, lets a 6-layer student match a 12-layer teacher on most GLUE tasks and RACE.

desk verdict Patient-KD is a simple, sensible extension of knowledge distillation to BERT compression, but the empirical case is built on single runs and small differences, so treat the headline numbers with caution. read the letter →

arxiv 1908.09355 v1 pith:67I7MYR5 submitted 2019-08-25 cs.CL cs.LG

Siqi Sun , Yu Cheng , Zhe Gan , Jingjing Liu This is my paper · ORCID
classification cs.CLcs.LG
keywords knowledgedistillationBERTcompressionmodelintermediatelayertransformerGLUEbenchmarkpatienthiddenstatematching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Patient Knowledge Distillation (Patient-KD), a way to compress a 12-layer BERT teacher into 3- or 6-layer student models. Instead of matching only the teacher's final softmax logits, the student also matches L2-normalized [CLS] token hidden states from selected intermediate teacher layers. On seven GLUE tasks and RACE, the 6-layer patient student outperforms both direct fine-tuning and vanilla knowledge distillation on most datasets, while cutting inference time roughly in half; the 3-layer version gives up to a 2.4x parameter reduction and 3.7x speedup. The authors' central claim is that patient multi-layer distillation transfers more knowledge than last-layer-only distillation and avoids the early saturation they observe with vanilla KD.

What carries the argument

The proposed mechanism is the patient distillation loss $L_{PT}$ (Eq. 7), which computes the mean squared error between L2-normalized [CLS] token hidden states of the student and corresponding teacher layers, and adds it to the knowledge-distillation objective as $L_{PKD} = (1-\alpha)L_{CE} + \alpha L_{DS} + \beta L_{PT}$. Layer correspondences are chosen by one of two strategies: PKD-Skip, which pairs student layers with teacher layers every $k$ steps (e.g., {2,4,6,8,10} for a 12-layer teacher), and PKD-Last, which uses the last $k-1$ teacher layers (e.g., {7,...,11}). The loss is computed only on the [CLS] token to keep training cheap and avoid noise from matching all token positions. This mechanism is what lets the student 'patiently' extract knowledge from multiple depths instead of one final layer.

What would settle it

Train a 6-layer student with the same Patient-KD objective but scramble the layer mapping, matching student layers to randomly chosen teacher layers instead of {2,4,6,8,10}; if performance remains at the PKD level, the specific layer correspondence is not what drives the gain. Alternatively, replace the [CLS] hidden states with random fixed vectors in $L_{PT}$ and check whether the reported improvements over vanilla KD on MNLI and QNLI disappear.

Watch

Extended reading notes

Core claim

The central claim is that a shallow BERT student trained to imitate the teacher's hidden representations at multiple depths, in addition to its final predictions, learns more effectively than one trained on final logits alone. Concretely, the paper adds a mean-squared-error term between L2-normalized [CLS] embeddings of the student and selected teacher layers to the standard distillation objective. With a 6-layer student initialized from the first six teacher layers, this Patient-KD objective yields the best results among compared methods on five of seven GLUE tasks and on RACE, and the authors report that the student begins to plateau only later in training than a vanilla-KD student, which they interpret as better generalization. The paper also shows the approach is not tied to a specific teacher: using a 24-layer BERT teacher still gives improvements over vanilla KD even when the student architecture is not a perfect match.

Load-bearing premise

The load-bearing premise is that L2-normalized [CLS] hidden states from different layers of the teacher are semantically comparable to the student's layers, so that matching them transfers useful knowledge; the paper tests two hand-picked layer mappings but does not analyze whether the correspondence is meaningful.

Editorial extensions

If this is right

  • A 6-layer BERT student trained with Patient-KD matches or approaches the 12-layer teacher on datasets with more than 60k training samples, suggesting large-data tasks are where compression pays off.
  • The 3-layer student still performs close to the teacher on QQP, indicating that patient distillation can sometimes push compression to a quarter of the original depth.
  • Inference time scales almost linearly with layer count: the paper reports a 1.94x speedup for BERT6 and 3.73x for BERT3 over BERT12 on a Titan RTX GPU.
  • Patient-KD also improves over vanilla KD when the teacher is BERT-Large, so the benefit is not specific to the BERT-Base teacher.
  • The learning curves suggest that vanilla KD saturates on the dev set while Patient-KD keeps improving, which the paper attributes to reduced overfitting during distillation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gain comes from matching normalized [CLS] states, then the same loss should transfer to other transformer encoders such as RoBERTa or XLNet, which the paper names as future targets but does not test.
  • The layer-mapping question could be settled by a sweep over permutations of teacher layers; such a sweep would also indicate whether the student is learning a semantic alignment or merely a denoising signal.
  • Because the loss uses only [CLS] states, it may be less effective for token-level tasks like named-entity recognition or span extraction, where per-token hidden states carry the label-relevant information.
  • A testable extension is to use the patient loss during pre-training rather than only fine-tuning, which the authors list as future work; if it helps, it would suggest the intermediate layers contain generalizable knowledge beyond task-specific labels.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Patient Knowledge Distillation (PKD), a method for compressing BERT teacher models into shallower BERT students by adding a mean-squared-error loss between the L2-normalized [CLS] hidden states of selected teacher layers and corresponding student layers to the standard knowledge-distillation objective. Two layer-mapping strategies are introduced: PKD-Last, which uses the teacher's last k layers, and PKD-Skip, which uses every k-th layer. Experiments on seven GLUE tasks and RACE with BERT3 and BERT6 students show that PKD outperforms both direct fine-tuning and vanilla knowledge distillation on most tasks, with the PKD-Skip variant generally performing slightly better. The paper also reports storage and inference speedups, and analyzes the effect of using BERT-Large as the teacher.

Significance. If the reported empirical results are reliable, the proposed method is a simple and reusable technique for BERT compression: it requires only adding one extra MSE term to the standard KD loss and yields consistent gains over vanilla KD and fine-tuning on several benchmarks, including official GLUE test-server results. The paper clearly states the loss definitions, the training protocol, and the hyperparameter search spaces, and it evaluates on external leaderboards, which avoids circularity. The paper also includes useful analyses of training efficiency, speedup, and teacher-quality effects. However, the central empirical claim--that the small reported gains are real and not due to run-to-run variance--is not yet established because all main results are single runs without error bars or significance tests, and one of the seven GLUE tasks (MRPC) shows a difference in the opposite direction. The method's reliance on a manually specified layer mapping also lacks supporting analysis.

major comments (3)
  1. [Section 4.3, Table 1] All GLUE results in Table 1 are single test-server submissions with no measure of variance. The reported PKD-over-KD gains range from about +0.2 to +1.3 GLUE points, which is within the run-to-run variability commonly observed when fine-tuning BERT on these tasks. The paper should report means and standard deviations over at least 3-5 seeds for each configuration, or provide a statistical significance test, before the central claim that PKD consistently improves over KD can be accepted.
  2. [Section 4.3, MRPC paragraph] The MRPC row in Table 1 directly contradicts the paper's general claim: BERT6-PKD achieves 85.0/79.9 on the test set versus 86.2/80.6 for BERT6-KD, so PKD is worse on this task. The authors respond by repeating the dev-set experiment three times and reporting means (82.23, 82.84, 83.46), but they do not report standard deviations or any significance test. This does not establish that PKD is not worse on MRPC; at best it suggests the difference may be within noise. The discussion should either provide rigorous evidence that the test-set difference is not significant or explicitly acknowledge MRPC as an exception to the claimed superiority.
  3. [Section 3.2, Eq. (7), and Section 4.2] The proposed loss assumes that the L2-normalized [CLS] hidden states of teacher layer I_pt(j) are semantically comparable to those of student layer j. The paper tests two hand-picked mappings (I_pt = {2,4,6,8,10} for PKD-Skip and {7,8,9,10,11} for PKD-Last) but provides no analysis of whether these correspondences are meaningful, how sensitive the results are to the choice of I_pt, or whether the student initialization from the first six teacher layers interacts with the choice of I_pt. If the layer mapping is poorly aligned, the L_PT term could inject noise, and the small observed gains might not generalize to other tasks or architectures. The paper should include at least an ablation or sensitivity analysis over different layer mappings.
minor comments (5)
  1. [Abstract and Section 4.3] The abstract states that the method achieves improved results 'without sacrificing model accuracy,' but Table 1 shows that the 6-layer PKD student is below the BERT12 teacher on every task (e.g., SST-2 92.0 vs. 94.3, RTE 65.5 vs. 69.1). Clarify whether the comparison is against the teacher model or against the vanilla KD baseline.
  2. [Section 4.2] The text says 'We initialize BERT k with the first k layers of parameters from pre-trained BERT-Base' and then defines I_pt for PKD-Skip as {2,4,6,8,10} for a BERT12-to-BERT6 distillation. It is not explained why the student's layer 6 is matched to teacher layer 10 rather than layer 12, given that layer 12 is connected to the softmax. A brief explanation of the chosen mapping would help.
  3. [Equation (7)] The notation h^s_{i,j} and h^t_{i,I_pt(j)} is not explicitly defined before the equation. Define h^s and h^t and their dimensions to avoid confusion, especially since the same symbol h is used elsewhere for the output embedding.
  4. [Table 1] In the row for BERT3-FT, the MRPC cell reads '80.5/ 72.6' with an extra space after the slash, while other cells use '80.5/72.6'. This formatting inconsistency should be fixed.
  5. [Section 4.5, Table 5] The caption and text refer to 'BERT 6[Base]' and 'BERT 6[Large]', but the table header uses 'BERT6[Base]' and 'BERT6[Large]'. Use a consistent notation throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the PKD objective is defined independently of the external GLUE and RACE test metrics, and the reported gains are measured on held-out test benchmarks rather than constructed from the loss itself.

full rationale

The paper's central claim is that adding the intermediate-layer MSE term L_PT in Eq. (7) to the standard KD objective in Eq. (8) improves student accuracy. This is an empirical proposal, not a mathematical derivation whose output is its own input. The loss is defined on training data and on teacher hidden states, while the headline results in Tables 1 and 3 are obtained from the GLUE test server and the RACE leaderboard, which are external and not used to fit the loss or select among reported test predictions. Hyperparameters alpha, T, and beta are chosen by validation accuracy, which is standard model selection and does not make test results circular. The layer-correspondence choice I_pt is heuristic and hand-specified, but a heuristic assumption is not a circular reduction. There are no load-bearing self-citations: the method builds on external prior work such as Hinton et al. (2015), FitNets, and BERT, and no uniqueness theorem or author-derived constraint is invoked to force the result. The paper's own discussion of MRPC overfitting and the absence of multiple-seed error bars are legitimate correctness and robustness concerns, but they are not circularity. The empirical claim is self-contained: train with Eq. (8), evaluate on held-out benchmarks, and compare against baselines.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several tuned hyperparameters (alpha, T, beta, learning rate) and on the unproven assumption that normalized [CLS] hidden states are a transferable training signal. No new physical or architectural entities are introduced; PKD-Last and PKD-Skip are training strategies, not new model components.

free parameters (5)
  • alpha = 0.2, 0.5, or 0.7 per task (grid-searched on dev)
    Weight balancing task cross-entropy and distillation loss in Eq. (5); not predicted by theory, selected by validation accuracy.
  • temperature T = 5, 10, or 20 per task (grid-searched on dev)
    Softens teacher logits in Eq. (2); chosen by grid search, affects distillation signal.
  • beta = 10, 100, 500, or 1000 per task (grid-searched on dev)
    Weight of the patient intermediate-layer loss L_PT in Eq. (8); central to the new method and chosen by validation accuracy.
  • patient layer set I_pt = PKD-Skip: {2,4,6,8,10}; PKD-Last: {7,8,9,10,11} for BERT12 to BERT6
    Hand-selected layer alignment between teacher and student; not optimized, affects which representations are matched.
  • learning rate = 1e-5, 2e-5, or 5e-5 per task
    Standard optimizer hyperparameter selected on dev; results depend on it.
assumptions (4)
  • domain assumption A fine-tuned BERT teacher's soft predictions carry enough task knowledge to train a smaller student.
    The entire KD setup requires the teacher's probabilities to be a useful training target; this is assumed, not derived, in Section 3.1.
  • domain assumption L2-normalized [CLS] hidden states from different depths are comparable and their differences are a useful learning signal.
    Eq. (7) defines the patient loss on normalized [CLS] vectors; the paper does not justify why scale should be discarded or why matching directions transfers knowledge.
  • domain assumption The student, initialized with the first k teacher layers, can be aligned to arbitrary teacher layers through the I_pt mapping.
    Section 3.2 and Eq. (7) match student layer j to teacher layer I_pt(j) without studying whether the correspondence is semantically meaningful.
  • domain assumption All tasks can be modeled with a [CLS]-based classification head.
    The method only distills [CLS] representations; the transferability of [CLS] features across sentence-pair, single-sentence, and reading-comprehension tasks is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Patient Knowledge Distillation for BERT Model Compression." pith.science (2026). https://pith.science/paper/67I7MYR5

@misc{pith2026190809355,
  author       = {Pith},
  title        = {Pith review of: Patient Knowledge Distillation for BERT Model Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/67I7MYR5}},
  note         = {Machine review of arXiv:1908.09355}
}
abstract

Pre-trained language models such as BERT have proven to be highly effective for natural language processing (NLP) tasks. However, the high demand for computing resources in training such models hinders their application in practice. In order to alleviate this resource hunger in large-scale model training, we propose a Patient Knowledge Distillation approach to compress an original large model (teacher) into an equally-effective lightweight shallow network (student). Different from previous knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model patiently learns from multiple intermediate layers of the teacher model for incremental knowledge extraction, following two strategies: ($i$) PKD-Last: learning from the last $k$ layers; and ($ii$) PKD-Skip: learning from every $k$ layers. These two patient distillation schemes enable the exploitation of rich information in the teacher's hidden layers, and encourage the student model to patiently learn from and imitate the teacher through a multi-layer distillation process. Empirically, this translates into improved results on multiple NLP tasks with significant gain in training efficiency, without sacrificing model accuracy.

Figures

Figures reproduced from arXiv: 1908.09355 by the authors.

Figure 1
Figure 1. Model architecture of the proposed Patient [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Accuracy on the training and dev sets of QNLI and MNLI datasets, by directly applying vanilla knowl [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 22 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

    cs.CL 2019-08 conditional novelty 7.0 of 10

    Pre-training compact BERT models before distillation, called Pre-trained Distillation, outperforms pre-training plus fine-tuning, plain distillation, and more complex compression baselines.

  2. Distill Where the Student Goes: Teacher-Regularized RL for English-Evidence Cross-Lingual RAG

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Teacher-regularized on-policy RL with a reverse-KL anchor on student prefixes improves language consistency and evidence-grounded correctness for English-evidence cross-lingual RAG, acting as a safety net against larg...

  3. NITP: Next Implicit Token Prediction for LLM Pre-training

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    NITP augments standard next-token prediction with implicit semantic prediction in representation space using shallow-layer self-supervision, reporting consistent downstream gains on 0.5B-9B models including 5.7% on MM...

  4. MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistants

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Training small language models on intermediate-length reasoning chains from a merged mid-sized teacher assistant improves their math reasoning scores over direct distillation from a large teacher.

  5. GenRecal: Generation after Recalibration from Large to Small Vision-Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A learnable Recalibrator bridges different tokenizers so that small VLMs can distill knowledge from any large VLM, improving their benchmark scores.

  6. InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models

    cs.LG 2025-05 conditional novelty 6.0 of 10

    InfiFPO replaces the DPO reference model with a clipped, max-margin-selected fusion of source model sequence probabilities, lifting Phi-4's 11-benchmark average from 79.95 to 83.33.

  7. Optimizing Knowledge Distillation in Transformers: Enabling Multi-Head Attention without Alignment Barriers

    cs.CV 2025-02 reject novelty 6.0 of 10

    Squeezing-Heads Distillation mixes several teacher attention maps into one per-sample weighted map, enabling knowledge distillation between transformers with different head counts without extra parameters.

  8. MaintaAvatar: A Maintainable Avatar Based on Neural Radiance Fields by Continual Learning

    cs.CV 2025-02 conditional novelty 6.0 of 10

    MaintaAvatar continually adds new appearances to a NeRF human avatar from a few images per task and retains old appearances via replay, per-appearance triplanes, and pose distillation.

  9. BEEM: Boosting Performance of Early Exit DNNs using Multi-Exit Classifiers as Experts

    cs.LG 2025-02 conditional novelty 6.0 of 10

    BEEM aggregates weighted confidence from consistent neighboring exit classifiers, resetting on disagreement, and sets thresholds from validation error rates to accelerate early-exit inference.

  10. CURing Large Models: Compression via CUR Decomposition

    cs.LG 2025-01 conditional novelty 6.0 of 10

    CUR decomposition with WANDA-and-DEIM row/column selection compresses LLM weights quickly, and the linking matrix U can be fine-tuned as a PEFT-style healing step.

  11. MAS-Attention: Memory-Aware Stream Processing for Attention Acceleration on Resource-Constrained Edge Devices

    cs.DC 2024-11 conditional novelty 6.0 of 10

    MAS-Attention pipelines tiled MatMul and softmax workloads onto separate MAC and vector units of edge NPUs, achieving up to 2.75x simulated and 1.76x measured speedups over FLAT while keeping outputs exact.

  12. SoftLMs: Efficient Adaptive Low-Rank Approximation of Language Models using Soft-Thresholding Mechanism

    cs.LG 2024-11 conditional novelty 6.0 of 10

    A per-layer learnable soft-thresholding mechanism for SVD-based low-rank compression reduces language model parameters by 50% with roughly 1% average accuracy loss.

  13. Progressive$^2$: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Progressive^2 improves knowledge distillation under large teacher-student capacity gaps by progressively including teacher layers and gradually compressing the student through self-distillation rounds.

  14. Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Structural pruning with finetuning plus hidden-state distillation recovers most performance in multimodal LLMs, with 5% of training data sufficient at moderate compression levels.

  15. AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes

    cs.AI 2025-06 reject novelty 5.0 of 10

    AgentDistill distills agent capabilities without any training by having a teacher generate reusable MCP tool boxes that small-model students invoke at inference time.

  16. Olica: Efficient Structured Pruning of Large Language Models without Retraining

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Olica prunes LLMs without retraining by low-rank decomposing attention-layer matrix products and adding small linear corrections to pruned feed-forward layers, matching or beating retraining-based methods on several b...

  17. Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment

    cs.LG 2025-01 conditional novelty 5.0 of 10

    Temporal-wise logits distillation with ensemble self-distillation lets one SNN, trained at T=6, be deployed at T=1 through T=6 without retraining and with competitive accuracy.

  18. Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Models

    cs.CL 2024-11 conditional novelty 5.0 of 10

    DynSDPB fine-tunes small language models by self-distilling soft labels from the previous mini-batch, with dynamic per-sample temperature and loss weighting.

  19. ElastiFormer: Learned Redundancy Reduction in Transformer via Self-Distillation

    cs.LG 2024-11 conditional novelty 5.0 of 10

    A post-training routing method that uses self-distillation to let frozen pretrained Transformers process only a subset of parameters and tokens, cutting active compute by 20 to 50 percent.

  20. BnBERT-iPET: Sparse Few-Shot Language Modeling for Bengali via Lottery Ticket Pruning

    cs.LG 2026-08 reject novelty 4.0 of 10

    A 90%-pruned few-shot Bengali model is reported to rival larger baselines on some tasks, but the reported F1 scores contradict the paper's own precision and recall values.

  21. Enhancing deep learning models for time series classification via knowledge distillation

    cs.LG 2026-07 conditional novelty 4.0 of 10

    Knowledge distillation most benefits intermediate-complexity students for time series classification, cutting parameters sharply while matching teacher accuracy across FCN, Inception, and ConvTran on UCR.

  22. TinyMusician: On-Device Music Generation with Knowledge Distillation and Mixed Precision Quantization

    cs.SD 2025-08 reject novelty 4.0 of 10

    TinyMusician distills MusicGen and applies hand-picked mixed-precision quantization to make a 1.04 GB on-device music generator, but the headline '93% quality, 55% smaller' claims conflict with the paper's own tables.

Reference graph

Works this paper leans on

42 extracted references · 20 canonical work pages · cited by 22 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL

    2017

  4. [4]

    Soravit Changpinyo, Mark Sandler, and Andrey Zhmoginov. 2017. The power of sparsity in convolutional neural networks. arXiv preprint arXiv:1702.06257

    arXiv 2017

  5. [5]

    Goodfellow, and Jonathon Shlens

    Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. 2015. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641

    arXiv 2015

  6. [6]

    Yu, Rogerio S

    Yu Cheng, Felix X. Yu, Rogerio S. Feris, Sanjiv Kumar, Alok Choudhary, and Shi-Fu Chang. 2015. An exploration of parameter redundancy in deep networks with circulant projections. In ICCV

    work page 2015

  7. [7]

    Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In EMNLP

    work page 2017

  8. [8]

    Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In NIPS

    work page 2015

Show all 42 references
  1. [9]

    Misha Denil, Babak Shakibi, Laurent Dinh, Marc'Aurelio Ranzato, and Nando de Freitas. 2013. Predicting parameters in deep learning. In NIPS

    2013

  2. [10]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  3. [11]

    William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing

    2005

  4. [12]

    Yunchao Gong, Liu Liu, Ming Yang, and Lubomir D. Bourdev. 2014. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115

    2014 arXiv

  5. [13]

    Song Han, Huizi Mao, and William J Dally. 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR

    2016

  6. [14]

    Song Han, Jeff Pool, John Tran, and William J. Dally. 2015. Learning both weights and connections for efficient neural networks. In NIPS

    2015

  7. [15]

    Yihui He, Xiangyu Zhang, and Jian Sun. 2017. Channel pruning for accelerating very deep neural networks. In ICCV

    2017

  8. [16]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    2015 arXiv

  9. [17]

    Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL

    2018

  10. [18]

    Weinberger

    Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. 2016. Deep networks with stochastic depth. In ECCV

    2016

  11. [19]

    Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In NIPS

    2015

  12. [20]

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683

    2017 arXiv

  13. [21]

    Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019 a . Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482

    2019 arXiv

  14. [22]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 b . Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692

    2019 arXiv

  15. [23]

    Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In ICLR

    2018

  16. [24]

    Liang Lu, Michelle Guo, and Steve Renals. 2017. Knowledge distillation for small-footprint highway networks. In ICASSP

    2017

  17. [25]

    Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NIPS

    2017

  18. [26]

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS

    2013

  19. [27]

    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP

    2014

  20. [28]

    Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL

    2018

  21. [29]

    Antonio Polino, Razvan Pascanu, and Dan Alistarh. 2018. Model compression via distillation and quantization. arXiv preprint arXiv:1802.05668

    2018 arXiv

  22. [30]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. arXiv

    2018

  23. [31]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP

    2016

  24. [32]

    Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2015. Fitnets: Hints for thin deep nets. In ICLR

    2015

  25. [33]

    Bharat Bhusan Sau and Vineeth N Balasubramanian. 2016. Deep model compression: Distilling knowledge from noisy teachers. arXiv preprint arXiv:1610.09650

    2016 arXiv

  26. [34]

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP

    2013

  27. [35]

    Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2019. Multilingual neural machine translation with knowledge distillation. In ICLR

    2019

  28. [36]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS

    2017

  29. [37]

    Vetrov, Jonathan Huang, Li Zhang, Maxwell Collins, Michael Figurnov, Ruslan Salakhutdinov, and Yukun Zhu

    Dmitry P. Vetrov, Jonathan Huang, Li Zhang, Maxwell Collins, Michael Figurnov, Ruslan Salakhutdinov, and Yukun Zhu. 2017. Spatially adaptive computation time for residual networks. In CVPR

    2017

  30. [38]

    Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461

    2018 arXiv

  31. [39]

    Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426

    2017 arXiv

  32. [40]

    Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237

    2019 arXiv

  33. [41]

    Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Mark Zhang. 2016. Doubly convolutional neural networks. In NIPS

    2016

  34. [42]

    Chenguang Zhu, Michael Zeng, and Xuedong Huang. 2018. Sdnet: Contextualized attention-based deep network for conversational question answering. arXiv preprint arXiv:1812.03593

    2018 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.