RECORD
Distilled Neural Networks for Efficient Learning to Rank
Machine Learning · Submitted 22 Feb 2022
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Comprehensive experiments on two public learning-to-rank datasets show that neural networks produced with our novel approach are competitive at any point of the effectiveness-efficiency trade-off when compared with tree-based ensembles, providing up to 4x scoring time speed-up without affecting the ranking quality.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Recent studies in Learning to Rank have shown the possibility to effectively distill a neural network from an ensemble of regression trees. This result leads neural networks to become a natural competitor of tree-based ensembles on the ranking task. Nevertheless, ensembles of regression trees outperform neural models both in terms of efficiency and effectiveness, particularly when scoring on CPU. In this paper, we propose an approach for speeding up neural scoring time by applying a combination of Distillation, Pruning and Fast Matrix multiplication. We employ knowledge distillation to learn shallow neural networks from an ensemble of regression trees. Then, we exploit an efficiency-oriented pruning technique that performs a sparsification of the most computationally-intensive layers of the neural network that is then scored with optimized sparse matrix multiplication. Moreover, by studying both dense and sparse high performance matrix multiplication, we develop a scoring time prediction model which helps in devising neural network architectures that match the desired efficiency requirements. Comprehensive experiments on two public learning-to-rank datasets show that neural networks produced with our novel approach are competitive at any point of the effectiveness-efficiency trade-off when compared with tree-based ensembles, providing up to 4x scoring time speed-up without affecting the ranking quality.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Recent studies in Learning to Rank have shown the possibility to effectively distill a neural network from an ensemble of regression trees. | background 1.00 | 0.04 | 0.02 | hedged possibility 0.79 |
| s1 | This result leads neural networks to become a natural competitor of tree-based ensembles on the ranking task. | interpretation 0.64 | 0.08 | 0.04 | asserted 0.90 |
| s2 | Nevertheless, ensembles of regression trees outperform neural models both in terms of efficiency and effectiveness, particularly when scoring on CPU. | background 0.61 | 0.14 | 0.04 | asserted 1.00 |
| s3 | In this paper, we propose an approach for speeding up neural scoring time by applying a combination of Distillation, Pruning and Fast Matrix multiplication. | contribution 0.93 | 0.76 | 0.06 | asserted 0.98 |
| s4 | We employ knowledge distillation to learn shallow neural networks from an ensemble of regression trees. | method 0.95 | 0.89 | 0.03 | asserted 0.99 |
| s5 | Then, we exploit an efficiency-oriented pruning technique that performs a sparsification of the most computationally-intensive layers of the neural network that is then scored with optimized sparse matrix multiplication. | method 0.99 | 0.93 | 0.03 | asserted 0.99 |
| s6 | Moreover, by studying both dense and sparse high performance matrix multiplication, we develop a scoring time prediction model which helps in devising neural network architectures that match the desired efficiency requirements. | method 0.56 | 0.95 | 0.04 | asserted 1.00 |
| s7 | Comprehensive experiments on two public learning-to-rank datasets show that neural networks produced with our novel approach are competitive at any point of the effectiveness-efficiency trade-off when compared with tree-based ensembles, providing up to 4x scoring time speed-up without affecting the ranking quality. | result 1.00 | 0.97 | 0.92 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Recent studies in Learning to Rank have shown the possibility to effectively distill a neural network from an ensemble of regression trees. This result leads neural networks to become a natural competitor of tree-based ensembles on the ranking task. Nevertheless, ensembles of regression trees outperform neural models both in terms of efficiency and effectiveness, particularly when scoring on CPU. In this paper, we propose an approach for speeding up neural scoring time by applying a combination of Distillation, Pruning and Fast Matrix multiplication. We employ knowledge distillation to learn shallow neural networks from an ensemble of regression trees. Then, we exploit an efficiency-oriented pruning technique that performs a sparsification of the most computationally-intensive layers of the neural network that is then scored with optimized sparse matrix multiplication. Moreover, by studying both dense and sparse high performance matrix multiplication, we develop a scoring time prediction model which helps in devising neural network architectures that match the desired efficiency requirements. Comprehensive experiments on two public learning-to-rank datasets show that neural networks produced with our novel approach are competitive at any point of the effectiveness-efficiency trade-off when compared with tree-based ensembles, providing up to 4x scoring time speed-up without affecting the ranking quality.
Human review
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites
Cites as background 2 papers
-
In MC techniques, we observe the presence of several lines of research: pruning [ref] , quantization [ref] design of efficient architectures [ref] , knowledge distillation [ref] .
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications · Model Compression · 0.96
-
The current state-of-the-art algorithm for DMM is the the well-known Goto Algorithm [ref] , on which are based several open (GotoBLAS [ref] , OpenBLAS [ref] , BLIS [ref] ) or commercial (Intel MKL [ref] ) implementations.
BLISlab: A Sandbox for Optimizing GEMM · Dense Matrix Multiplication · 0.91
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.