A cost-aware value-of-information acquisition function is derived to balance direct observations against noisy pairwise human comparisons in Bayesian optimization, approaching the convex hull of the individual information sources' performance trajectories.
When is it better to compare than to score?
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
abstract
When eliciting judgements from humans for an unknown quantity, one often has the choice of making direct-scoring (cardinal) or comparative (ordinal) measurements. In this paper we study the relative merits of either choice, providing empirical and theoretical guidelines for the selection of a measurement scheme. We provide empirical evidence based on experiments on Amazon Mechanical Turk that in a variety of tasks, (pairwise-comparative) ordinal measurements have lower per sample noise and are typically faster to elicit than cardinal ones. Ordinal measurements however typically provide less information. We then consider the popular Thurstone and Bradley-Terry-Luce (BTL) models for ordinal measurements and characterize the minimax error rates for estimating the unknown quantity. We compare these minimax error rates to those under cardinal measurement models and quantify for what noise levels ordinal measurements are better. Finally, we revisit the data collected from our experiments and show that fitting these models confirms this prediction: for tasks where the noise in ordinal measurements is sufficiently low, the ordinal approach results in smaller errors in the estimation.
years
2026 2verdicts
UNVERDICTED 2representative citing papers
Base Score Extraction Functions convert user preferences into base scores for Bipolar Argumentation Frameworks, producing Quantitative Bipolar Argumentation Frameworks usable with existing gradual semantics tools, including an algorithm and robotics evaluation.
citing papers explorer
-
Elicitation-Augmented Bayesian Optimization
A cost-aware value-of-information acquisition function is derived to balance direct observations against noisy pairwise human comparisons in Bayesian optimization, approaching the convex hull of the individual information sources' performance trajectories.
-
From User Preferences to Base Score Extraction Functions in Gradual Argumentation (with Appendix)
Base Score Extraction Functions convert user preferences into base scores for Bipolar Argumentation Frameworks, producing Quantitative Bipolar Argumentation Frameworks usable with existing gradual semantics tools, including an algorithm and robotics evaluation.