Curriculum Learning Strategies for IR: An Empirical Study on Conversation Response Ranking

Claudia Hauff; Gustavo Penha

RECORD

Curriculum Learning Strategies for IR: An Empirical Study on Conversation Response Ranking

Gustavo Penha, Claudia Hauff

Information Retrieval · Submitted 18 Dec 2019

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim We find that, overall, by just intelligently sorting the training data (i.e., by performing curriculum learning) we can improve the retrieval effectiveness by up to 2%.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID1912.08555 (version 1)

Pith IDUHRZJ7QD, Pith's permanent record of this paper and its review

CitationClaudia Hauff, Gustavo Penha. "Curriculum Learning Strategies for IR: An Empirical Study on Conversation Response Ranking." arXiv:1912.08555 (2019). https://pith.science/paper/UHRZJ7QD

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,275 input tokens.

What the abstract itself states, as probabilities: claims first 0.59 · states limitation 0.29 · human participants 0.06 · controlled comparison 0.78 · randomized assignment 0.04 · states quantitative result 0.98 · compares with prior results 0.71 · mentions code or data availability 0.02.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; show every role.

Neural ranking models are traditionally trained on a series of random batches, sampled uniformly from the entire training set. Curriculum learning has recently been shown to improve neural models' effectiveness by sampling batches non-uniformly, going from easy to difficult instances during training. In the context of neural Information Retrieval (IR) curriculum learning has not been explored yet, and so it remains unclear (1) how to measure the difficulty of training instances and (2) how to transition from easy to difficult instances during training. To address both challenges and determine whether curriculum learning is beneficial for neural ranking models, we need large-scale datasets and a retrieval task that allows us to conduct a wide range of experiments. For this purpose, we resort to the task of conversation response ranking: ranking responses given the conversation history. In order to deal with challenge (1), we explore scoring functions to measure the difficulty of conversations based on different input spaces. To address challenge (2) we evaluate different pacing functions, which determine the velocity in which we go from easy to difficult instances. We find that, overall, by just intelligently sorting the training data (i.e., by performing curriculum learning) we can improve the retrieval effectiveness by up to 2%.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Neural ranking models are traditionally trained on a series of random batches, sampled uniformly from the entire training set.	background 1.00	0.05	0.01	asserted 1.00
s1	Curriculum learning has recently been shown to improve neural models' effectiveness by sampling batches non-uniformly, going from easy to difficult instances during training.	background 1.00	0.05	0.03	asserted 0.99
s2	In the context of neural Information Retrieval (IR) curriculum learning has not been explored yet, and so it remains unclear (1) how to measure the difficulty of training instances and (2) how to transition from easy to difficult instances during training.	gap 1.00	0.05	0.02	asserted 0.87
s3	To address both challenges and determine whether curriculum learning is beneficial for neural ranking models, we need large-scale datasets and a retrieval task that allows us to conduct a wide range of experiments.	method 0.54	0.18	0.02	asserted 0.91
s4	For this purpose, we resort to the task of conversation response ranking: ranking responses given the conversation history.	method 0.73	0.73	0.02	asserted 0.89
s5	In order to deal with challenge (1), we explore scoring functions to measure the difficulty of conversations based on different input spaces.	method 0.89	0.88	0.02	asserted 0.81
s6	To address challenge (2) we evaluate different pacing functions, which determine the velocity in which we go from easy to difficult instances.	method 0.93	0.87	0.02	asserted 0.62
s7	We find that, overall, by just intelligently sorting the training data (i.e., by performing curriculum learning) we can improve the retrieval effectiveness by up to 2%.	result 1.00	0.97	0.93	asserted 1.00

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.99.

What was examined, and what was not

sources read

arXiv listing abstract v1

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

Neural ranking models are traditionally trained on a series of random batches, sampled uniformly from the entire training set. Curriculum learning has recently been shown to improve neural models' effectiveness by sampling batches non-uniformly, going from easy to difficult instances during training. In the context of neural Information Retrieval (IR) curriculum learning has not been explored yet, and so it remains unclear (1) how to measure the difficulty of training instances and (2) how to transition from easy to difficult instances during training. To address both challenges and determine whether curriculum learning is beneficial for neural ranking models, we need large-scale datasets and a retrieval task that allows us to conduct a wide range of experiments. For this purpose, we resort to the task of conversation response ranking: ranking responses given the conversation history. In order to deal with challenge (1), we explore scoring functions to measure the difficulty of conversations based on different input spaces. To address challenge (2) we evaluate different pacing functions, which determine the velocity in which we go from easy to difficult instances. We find that, overall, by just intelligently sorting the training data (i.e., by performing curriculum learning) we can improve the retrieval effectiveness by up to 2%.

Human review

How Pith checks papers → · Full audit of this paper →

References

How the literature uses it

9 arXiv papers cite this in 21 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 16 sentences without a confident label

Builds on it 1 paper

Then, we can modify the definition of the pacing function in [ref] as follows: [equation] With the above pacing function, the parameter of $θ$ would increase linearly as the training process.

Adversarial Curriculum Graph Contrastive Learning with Pair-wise Augmentation · Adversarial Curriculum Learning · 0.81

Uses its method or tool 1 paper

For training scheduling, we apply the widely used predefined pacing function in our framework [ref] .

Adversarial Curriculum Graph Contrastive Learning with Pair-wise Augmentation · Adversarial Curriculum Learning · 0.96

Cites it as support 1 paper

This observation is inline with the findings from [ref] and [ref] on different tasks, where raising rapidly the self-paced rate in the first half of the training benefits the generalization performance.

Self-Paced Contrastive Learning for Semi-supervised Medical Image Segmentation with Meta-labels · Hard vs. linear self-paced regularization · 0.99

Cites it as background 2 papers

A popular approach to conversational search is retrieval-based [ref] : given an ongoing conversation and a large corpus of historic conversations, retrieve the response that is best suited from the corpus [ref] .

Do the Findings of Document and Passage Retrieval Generalize to the Retrieval of Responses for Dialogues? · Introduction · 0.95
A popular approach to conversational search is retrieval-based [ref] : given an ongoing conversation and a large corpus of historic conversations, retrieve the response that is best suited from the corpus (i.e., conversation response ranking [ref] ).

Sparse and Dense Approaches for the Full-rank Retrieval of Responses for Dialogues · Introduction · 0.92

What it cites · 9 sentences without a confident label

Uses the method or tool of 3 papers

$∙$ $(U,R,Y)$ and $(U,R,Y)$ : Inspired by CL literature [ref] , we use external model prediction confidence scores as a measure of difficulty[We note, that using BM25 average precision as a scoring function failed to outperform the baseline.].

On The Power of Curriculum Learning in Training Deep Networks · Scoring Functions · 0.84
The $step$ function [ref] divides the data into $S$ fixed sized groups, and after $T/S$ iterations a new group of instances is added, where $S$ is a hyperparameter.

Image Difficulty Curriculum for Generative Adversarial Networks (CuGAN) · Pacing functions · 0.85
We use cross entropy loss and the Adam optimizer [ref] with learning rate of $5e-5$ and $ϵ= 1e-8$.

Adam: A Method for Stochastic Optimization · Implementation Details · 1.00

Uses data or values from 1 paper

[ is available at <https://guzpenha.github.io/MANtIS/>] [ref] was created by us and contains 1.3 million context-response pairs built from conversations of 14 different sites of .

Introducing MANtIS: a novel Multi-Domain Information Seeking Dialogues Dataset · Datasets · 0.92

Compares with 1 paper

Our results bear resemblance to Born Again Networks [ref] , where a student model which is identical in parameters and architecture to the teacher model outperforms the teacher when trained with knowledge distillation [ref] , i.e., using the predictions of the teacher model as labels for the student model.

Distilling the Knowledge in a Neural Network · Scoring Functions · 0.88

Cites as background 4 papers

In a short time though, BERT [ref] (released in late 2018) and its derivatives (e.g. XLNet [ref] , RoBERTa [ref] ) have proven to be remarkably effective for a range of NLP tasks.

XLNet: Generalized Autoregressive Pretraining for Language Understanding · Introduction · 0.88
In a short time though, BERT [ref] (released in late 2018) and its derivatives (e.g. XLNet [ref] , RoBERTa [ref] ) have proven to be remarkably effective for a range of NLP tasks.

RoBERTa: A Robustly Optimized BERT Pretraining Approach · Introduction · 0.88
The BERT approach to learn textual representations has also significantly improved the performance of neural models for several IR tasks [ref] , that for a long time struggled to outperform classic IR models [ref] .

Simple Applications of BERT for Ad Hoc Document Retrieval · Neural Ranking Models · 0.81

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.