Knowledge Distillation via Weighted Ensemble of Teaching Assistants

Durga Prasad Ganta; Himel Das Gupta; Victor S. Sheng

RECORD

Knowledge Distillation via Weighted Ensemble of Teaching Assistants

Durga Prasad Ganta, Himel Das Gupta, Victor S. Sheng

Machine Learning · Submitted 23 Jun 2022

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim In this research, we have shown that using multiple teaching assistant models, the student model (the smaller model) can be further improved.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2206.12005 (version 1)

Pith ID2RS4D4ZG, Pith's permanent record of this paper and its review

CitationDurga Prasad Ganta, Himel Das Gupta, Victor S. Sheng. "Knowledge Distillation via Weighted Ensemble of Teaching Assistants." arXiv:2206.12005 (2022). https://pith.science/paper/2RS4D4ZG

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 4,901 input tokens.

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.09 · human participants 0.02 · controlled comparison 0.12 · randomized assignment 0.02 · states quantitative result 0.04 · compares with prior results 0.12 · mentions code or data availability 0.02.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; show every role.

Knowledge distillation in machine learning is the process of transferring knowledge from a large model called the teacher to a smaller model called the student. Knowledge distillation is one of the techniques to compress the large network (teacher) to a smaller network (student) that can be deployed in small devices such as mobile phones. When the network size gap between the teacher and student increases, the performance of the student network decreases. To solve this problem, an intermediate model is employed between the teacher model and the student model known as the teaching assistant model, which in turn bridges the gap between the teacher and the student. In this research, we have shown that using multiple teaching assistant models, the student model (the smaller model) can be further improved. We combined these multiple teaching assistant models using weighted ensemble learning where we have used a differential evaluation optimization algorithm to generate the weight values.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Knowledge distillation in machine learning is the process of transferring knowledge from a large model called the teacher to a smaller model called the student.	background 1.00	0.04	0.02	asserted 1.00
s1	Knowledge distillation is one of the techniques to compress the large network (teacher) to a smaller network (student) that can be deployed in small devices such as mobile phones.	background 1.00	0.05	0.03	asserted 1.00
s2	When the network size gap between the teacher and student increases, the performance of the student network decreases.	background 0.95	0.15	0.03	asserted 1.00
s3	To solve this problem, an intermediate model is employed between the teacher model and the student model known as the teaching assistant model, which in turn bridges the gap between the teacher and the student.	background 0.81	0.09	0.05	asserted 1.00
s4	In this research, we have shown that using multiple teaching assistant models, the student model (the smaller model) can be further improved.	contribution 0.60	0.94	0.76	asserted 1.00
s5	We combined these multiple teaching assistant models using weighted ensemble learning where we have used a differential evaluation optimization algorithm to generate the weight values.	method 0.98	0.95	0.19	asserted 0.98

What kind of paper the classifier reports this is: method or tool 0.99 · evidence from observation or experiment 0.68.

What was examined, and what was not

sources read

arXiv listing abstract v1

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

Knowledge distillation in machine learning is the process of transferring knowledge from a large model called the teacher to a smaller model called the student. Knowledge distillation is one of the techniques to compress the large network (teacher) to a smaller network (student) that can be deployed in small devices such as mobile phones. When the network size gap between the teacher and student increases, the performance of the student network decreases. To solve this problem, an intermediate model is employed between the teacher model and the student model known as the teaching assistant model, which in turn bridges the gap between the teacher and the student. In this research, we have shown that using multiple teaching assistant models, the student model (the smaller model) can be further improved. We combined these multiple teaching assistant models using weighted ensemble learning where we have used a differential evaluation optimization algorithm to generate the weight values.

Human review

How Pith checks papers → · Full audit of this paper →

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.