RECORD
Data vs classifiers, who wins?
Machine Learning · Submitted 15 Jul 2021
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Regarding the algorithms, the methodology proposed herein identified the Random Forest as the algorithm with the best innate ability.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
The experiments covered by Machine Learning (ML) must consider two important aspects to assess the performance of a model: datasets and algorithms. Robust benchmarks are needed to evaluate the best classifiers. For this, one can adopt gold standard benchmarks available in public repositories. However, it is common not to consider the complexity of the dataset when evaluating. This work proposes a new assessment methodology based on the combination of Item Response Theory (IRT) and Glicko-2, a rating system mechanism generally adopted to assess the strength of players (e.g., chess). For each dataset in a benchmark, the IRT is used to estimate the ability of classifiers, where good classifiers have good predictions for the most difficult test instances. Tournaments are then run for each pair of classifiers so that Glicko-2 updates performance information such as rating value, rating deviation and volatility for each classifier. A case study was conducted hereby which adopted the OpenML-CC18 benchmark as the collection of datasets and pool of various classification algorithms for evaluation. Not all datasets were observed to be really useful for evaluating algorithms, where only 10% were considered really difficult. Furthermore, the existence of a subset containing only 50% of the original amount of OpenML-CC18 was verified, which is equally useful for algorithm evaluation. Regarding the algorithms, the methodology proposed herein identified the Random Forest as the algorithm with the best innate ability.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | The experiments covered by Machine Learning (ML) must consider two important aspects to assess the performance of a model: datasets and algorithms. | background 1.00 | 0.19 | 0.02 | asserted 0.99 |
| s1 | Robust benchmarks are needed to evaluate the best classifiers. | background 0.94 | 0.05 | 0.03 | asserted 0.94 |
| s2 | For this, one can adopt gold standard benchmarks available in public repositories. | background 0.88 | 0.07 | 0.02 | hedged possibility 0.91 |
| s3 | However, it is common not to consider the complexity of the dataset when evaluating. | gap 0.89 | 0.07 | 0.02 | asserted 0.95 |
| s4 | This work proposes a new assessment methodology based on the combination of Item Response Theory (IRT) and Glicko-2, a rating system mechanism generally adopted to assess the strength of players (e.g., chess). | contribution 0.88 | 0.68 | 0.07 | asserted 0.98 |
| s5 | For each dataset in a benchmark, the IRT is used to estimate the ability of classifiers, where good classifiers have good predictions for the most difficult test instances. | method 1.00 | 0.43 | 0.05 | asserted 0.99 |
| s6 | Tournaments are then run for each pair of classifiers so that Glicko-2 updates performance information such as rating value, rating deviation and volatility for each classifier. | method 1.00 | 0.65 | 0.03 | asserted 1.00 |
| s7 | A case study was conducted hereby which adopted the OpenML-CC18 benchmark as the collection of datasets and pool of various classification algorithms for evaluation. | method 0.87 | 0.92 | 0.03 | asserted 1.00 |
| s8 | Not all datasets were observed to be really useful for evaluating algorithms, where only 10% were considered really difficult. | result 1.00 | 0.88 | 0.12 | asserted 0.90 |
| s9 | Furthermore, the existence of a subset containing only 50% of the original amount of OpenML-CC18 was verified, which is equally useful for algorithm evaluation. | result 1.00 | 0.92 | 0.15 | asserted 0.99 |
| s10 | Regarding the algorithms, the methodology proposed herein identified the Random Forest as the algorithm with the best innate ability. | result 1.00 | 0.94 | 0.37 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v4
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
The experiments covered by Machine Learning (ML) must consider two important aspects to assess the performance of a model: datasets and algorithms. Robust benchmarks are needed to evaluate the best classifiers. For this, one can adopt gold standard benchmarks available in public repositories. However, it is common not to consider the complexity of the dataset when evaluating. This work proposes a new assessment methodology based on the combination of Item Response Theory (IRT) and Glicko-2, a rating system mechanism generally adopted to assess the strength of players (e.g., chess). For each dataset in a benchmark, the IRT is used to estimate the ability of classifiers, where good classifiers have good predictions for the most difficult test instances. Tournaments are then run for each pair of classifiers so that Glicko-2 updates performance information such as rating value, rating deviation and volatility for each classifier. A case study was conducted hereby which adopted the OpenML-CC18 benchmark as the collection of datasets and pool of various classification algorithms for evaluation. Not all datasets were observed to be really useful for evaluating algorithms, where only 10% were considered really difficult. Furthermore, the existence of a subset containing only 50% of the original amount of OpenML-CC18 was verified, which is equally useful for algorithm evaluation. Regarding the algorithms, the methodology proposed herein identified the Random Forest as the algorithm with the best innate ability.
Human review
How the literature uses it
1 arXiv paper cite this in 1 sentence
What it cites · 1 sentence without a confident label
Compares with 1 paper
-
And like Dynabench [ref] , this work aims to create and maintain quality benchmarks, evaluating their ability to test classifiers through IRT parameters.
Adversarial NLI: A New Benchmark for Natural Language Understanding · Comparison with Related Work · 0.85
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.