Pith. sign in

REVIEW 3 major objections 5 minor 68 references

What should an AI assessor optimise for?

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An AI assessor can rank a target metric's values better by training on a monotone proxy loss — logistic for regression, logarithmic for classification — than by training on the target metric itself.

desk verdict Useful proxy-loss result for assessors, with a clean classification finding and a regression claim that leaks the true mean error through an oracle scale B. read the letter →

arxiv 2502.00365 v1 pith:ZNXBUIQC submitted 2025-02-01 cs.LG cs.AI

Daniel Romero-Alvarado , Fernando Martínez-Plumed , José Hernández-Orallo This is my paper · ORCID
classification cs.LGcs.AI
keywords AIassessorsproxylossesmonotonictransformationslogisticlossproperscoringrulesrankingSpearmancorrelationinstance-levelperformanceprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

An AI assessor is a second model trained to predict, for each input example, the loss another model will incur on some chosen metric. This paper asks whether the assessor should always be trained on that target metric, or whether training it on a different proxy metric and then transforming its predictions can rank the target values better. Using 10 regression and 10 classification tabular datasets, each with 255 base model configurations, it reports that the intuitive choice is not generally the best one: the logistic loss beats direct training for predicting absolute and squared errors in regression, and the logarithmic score beats direct training for predicting quadratic and spherical scores in classification. If the finding holds, one assessor trained on a well-chosen proxy could serve a whole family of monotonically related metrics, and assessor design becomes a question of choosing the right learning signal rather than simply copying the evaluation metric.

What carries the argument

The machinery is a small family of loss and score functions with exact monotonic transformations between them. For regression the signed simple, squared, and logistic losses are $L_1^\mp(\hat y,y) = \hat y - y$, $L_2^\mp = (\hat y-y)|\hat y-y|$, and $L_L^\mp = 2/(1+e^{-B(\hat y-y)})-1$ with the scale constant $B=\ln 3/\mathrm{mean}_Y|\hat y-y|$; unsigned versions are their absolute values. For binary classification the logarithmic, quadratic, and spherical scores are $S_L=\ln r_{\circledcirc}$, $S_Q=2r_{\circledcirc}-r\cdot r$, and $S_S=r_{\circledcirc}/\|r\|$, where $r_{\circledcirc}$ is the probability assigned to the observed class, and the paper derives invertible formulas such as $S_S=e^{S_L}/\sqrt{2e^{2S_L}-2e^{S_L}+1}$ and $S_Q=-(2e^{2S_L}-4e^{S_L}+1)$. The assessor is trained to predict the proxy, and its output is pushed through the inverse transform to the target metric. The explanatory mechanism is that the logistic loss saturates on extreme residuals, counteracting the double penalisation of squared error, while the logarithmic score amplifies rare confident errors that the other scores underweight.

What would settle it

Run the paper's exact 10-regression-dataset protocol but estimate each base model's $B$ from a held-out subset of residuals instead of using the full test-set mean absolute error; if the logistic-proxy assessor no longer beats the target-trained assessor in a majority of datasets, the headline claim depends on test-set knowledge the assessor would not have in practice.

Watch

Extended reading notes

Core claim

The paper's central claim is that for an assessor, the loss used in training and the loss one wants to estimate need not coincide: training on a different, monotonically related metric and inverting the transformation can give better instance-level ranking of the target metric. In regression, the unsigned logistic loss used as a proxy beats the target-trained assessor for unsigned simple error and unsigned squared error on net scores of 3 and 8 out of 10 datasets, and the signed logistic loss likewise beats direct training for the signed versions (4 and 7 out of 10). In classification, the logarithmic score outperforms direct training for the quadratic score in 8 of 10 datasets and for the spherical score in 9 of 10, while the quadratic score beats direct training for the spherical score in 7 of 10. The paper also reports the opposite side: signed losses, and signed squared error in particular, are poor proxies for unsigned targets because assessor predictions regress toward zero and lose magnitude information. Performance throughout is measured by Spearman rank correlation between predicted and true metric values, with bootstrap confidence intervals deciding wins, ties, and losses.

Load-bearing premise

The regression result depends on knowing the base model's average absolute error on the assessed data, because the logistic loss's scale constant $B$ is defined from that average and the inverse transformations reuse it; if an assessor must estimate $B$ rather than read it from the test set, the logistic-proxy advantage may shrink or disappear.

Editorial extensions

If this is right

  • Assessor training can be decoupled from the deployment metric: a single proxy-trained assessor can be transformed to any monotonically related target, reducing the need to retrain per evaluation metric.
  • Signed losses, especially signed squared error, should be avoided as proxies for unsigned targets, because mean reversion compresses predictions toward zero and underestimates large losses.
  • Proxy advantages are transitive over the studied loss families, so composing monotonic transformations preserves or propagates the ranking benefit.
  • The proxy effect holds across four assessor model classes, so it is not an artifact of one regressor's inductive bias.
  • The result opens a route to estimating composite metrics like toxicity or F1 by training separate assessors on monotonic components and integrating their predictions, as the paper itself suggests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the logistic-loss route needs the base model's mean absolute error to set the scale constant $B$ and to invert the transformation, a deployment that can only estimate $B$ from a labelled validation set may see a smaller or absent advantage; testing under that information constraint is a natural next step.
  • The curvature-matching explanation suggests a transferable recipe: for convex target losses choose a saturating proxy, and for under-penalising scores choose a steep proxy; this recipe could be tested on losses beyond the six studied here.
  • Since the paper evaluates ranking rather than calibrated prediction, the proxy advantage should also show up in downstream decisions that order instances by expected error, such as routing, rejection, or delegation, where only relative order matters.
  • An extension to structured outputs (translation, generation, toxicity) could reveal whether a single universal proxy metric exists across otherwise incomparable evaluation families.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper asks whether an AI assessor—a model that predicts the loss or score that another 'subject' model will incur on a given instance—should be trained on the target metric or on a proxy metric followed by a transformation. Using 10 regression and 10 classification tabular datasets, with 255 tree-based subject models per dataset, the authors compare direct target training against proxy training for signed/unsigned simple, squared, and logistic losses (regression) and logarithmic, quadratic, and spherical scores (classification). They derive exact monotonic transformations among the classification scores and report that, for Spearman correlation, the logistic loss is often a better proxy for absolute/squared regression errors than the target loss itself, and the logarithmic score is often a better proxy for quadratic/spherical scores. They conclude that training assessors directly on the target metric is not always optimal.

Significance. The question is novel and timely, and the empirical protocol has real strengths: instance-level splitting prevents leakage, 255 heterogeneous subject models per dataset give broad coverage, four assessor families are used, bootstrap confidence intervals support the win/tie/loss counts, and the dataset collection is standard and reproducible. The classification transformations (Eqs. 9-14) are exact, parameter-free derivations, which makes the logarithmic-score advantage in classification a robust and publishable finding. The regression route, by contrast, depends on a scale constant B that is an aggregate statistic of the target loss itself, and the evaluation is limited to Spearman correlation, so the paper's headline claim that the logistic loss is 'useful for minimising absolute or quadratic errors' is not yet supported. If the regression concern is resolved, the paper would be a useful counterpoint to the default practice of training assessors directly on the target metric.

major comments (3)
  1. [Definition 3.3, Eqs. (3), (28), (37), (39)-(40)] The logistic proxy route is not independent of the target loss. B = ln 3 / mean_Y |ŷ − y| (Eq. 3) is the mean absolute error of the subject model over the assessed data, and the inverse maps from logistic to simple and squared losses (Eqs. 28, 37, 39, and 40) reuse exactly this B. The proxy assessor therefore receives, through B, an aggregate statistic of the target loss that the direct target assessor never sees. In the deployment scenario the paper motivates, this mean absolute error is unknown—it is precisely the quantity the assessor is supposed to predict. The text also does not specify whether B is computed per subject model or pooled over all 255 models in a dataset, which is essential for reproducing the results. Please run an additional experiment where B is estimated from the assessor's training data only, or treated as a feature, and report Spearman margins and a calibration measure as a function of B misspecification; the current comparison is asymmetric.
  2. [Section 4 (Evaluation metrics), Section 5] The only evaluation criterion is Spearman's rank correlation between the predicted and true target loss. Spearman is invariant under monotonic transformations of the prediction, so it does not establish that the proxy-plus-inverse route 'minimises' absolute or quadratic errors, as the abstract claims. It is entirely possible for the proxy route to exhibit higher rank correlation while being badly biased or miscalibrated in value. Please report point-accuracy and calibration metrics for the final predicted target loss—for example, MAE/RMSE against the true target losses, bias, and reliability diagrams—for both the target and proxy assessors, and show the win/tie/loss counts for these metrics as well.
  3. [Section 4 (Assessor models) and Appendix A.2.1 (Eqs. 36-40)] The inverse transformations from logistic loss to simple/squared losses are only defined for |L∓_L| < 1 because the logarithm is taken of (1 ± L∓_L)/(1 ∓ L∓_L). Linear regression, Bayesian ridge, and feed-forward neural network assessors can output values outside [-1, 1] when trained on the signed logistic loss (or outside [0, 1] for the unsigned version). The paper does not state how such out-of-range predictions are handled (clipping, truncation, or rejection). This is a reproducibility issue and may affect the regression results for three of the four assessor families. Please specify the exact post-processing used and, if clipping is applied, show its effect.
minor comments (5)
  1. [Section 2.2] The phrase 'as a a distribution' should read 'as a distribution'.
  2. [Section 3] "mononotically related" should be "monotonically related".
  3. [Figure 16 caption] The caption labels the right panel as "Aggregated Spearman margin matrix for XGBoost assessor model", which appears to be copy-pasted from Figure 4; clarify whether the right panel is the aggregate over all assessor models or only XGBoost.
  4. [Appendix C] "Auction Verificatoin" should be "Auction Verification".
  5. [Section 5.1] The statement "This property holds for all pairs of losses in the diagram" is stronger than what the arrow-based analysis demonstrates; please soften the claim or provide a systematic check for all pairs rather than only the arrows shown in Figure 7.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the proxy-vs-target comparisons are empirical rank-based evaluations; the regression scale constant B is rank-invariant and the classification score maps are parameter-free algebraic inverses.

full rationale

The paper's central claim is empirical: for a given assessor model class and dataset, does training on a proxy loss followed by a monotone transformation yield a better Spearman rank correlation with the target loss than training directly on the target loss? No step in this chain reduces to its inputs by construction. The most suspicious candidate is the regression scale constant B in Definition 3.3: B = ln 3 / mean_Y |ŷ − y|, which is computed from the mean absolute residual of the base model and reused in the inverse maps in Appendix A.2.1 (e.g., L+1 = (1/B)|ln((1+L_L)/(1−L_L))| and L+2 = (1/B²)[ln((1+L_L)/(1−L_L))]²). However, B is a single positive constant per dataset, and both inverse maps are strictly increasing functions of the proxy prediction. Spearman's ρ is invariant under strictly increasing transformations, so the value of B cannot manufacture or force the reported correlation advantage: ρ(f(ˆP), T) = ρ(ˆP, T) for any fixed positive B. The proxy route therefore tests a genuine empirical question: whether a regressor trained on tanh-scaled residual magnitudes ranks target losses better than one trained on |e| or e² directly. The classification transformations (Eqs. 9–14) are exact algebraic identities derived from the definitions of SL, SQ and SS under the binary-class assumption. They contain no fitted constants, no learned parameters, and no hidden dependence on the target score values beyond the score definitions themselves. Using them to map a predicted logarithmic score to a quadratic or spherical score is therefore a compositional identity, not a circular prediction. The paper does cite earlier work by the same group for the notion of assessors (Hernández-Orallo et al., 2022), but that citation is background context rather than load-bearing: the experimental methodology, the datasets, the assessor models, and the Spearman-margin comparisons are new and self-contained against external benchmark data. No uniqueness theorem or ansatz is imported from the authors' prior work to forbid alternatives. A remaining methodological concern is that B is computed from the full dataset, including the test portion, which is a potential leakage or robustness issue if the goal were to predict absolute loss values. But because the evaluation metric is Spearman rank correlation and B is rank-invariant, this concern does not amount to circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 1 invented entities

The central claim rests on three categories: a fitted scale B in the logistic loss definition; assumptions that monotonic transformations are legitimate and that Spearman correlation measures assessor quality; and the new signed logistic loss as an introduced construct. The paper's strongest regression result depends on B, which is estimated from the target errors, while the classification result is more self-contained.

free parameters (1)
  • Logistic loss scale B = ln(3) / mean_Y |y_hat - y|, computed per base model from the target data
    Used to define the signed logistic error in Eq. (3) and reused in the inverse transforms in Eqs. (28) and (40). Because B is fitted to the same target errors the assessor predicts, it can inflate the stated advantage of the logistic proxy.
assumptions (3)
  • standard math The loss functions L1, L2, and LL are monotonic functions of the residual and therefore mutually transformable.
    The paper derives these transformations in Appendix A. This justifies mapping proxy predictions to target losses.
  • domain assumption Spearman rank correlation is an appropriate measure of assessor quality.
    The paper evaluates all assessors by Spearman rho between predicted and true target losses. It never validates predictive calibration or absolute error of the final loss estimates, so the claim that proxies are 'useful for minimising losses' rests on this unstated assumption.
  • domain assumption Tree-based base models and 10 datasets per setting are representative enough to support general claims about assessors.
    The conclusion that some monotonic transformations are promising is inferred from 10 datasets per task and 5 tree algorithms. No theory or broader benchmark guarantees generalization beyond these settings.
invented entities (1)
  • Signed logistic error LLL (L with subscript L)
    purpose: A smooth, saturating loss introduced as a proxy for simple and squared errors in regression.
    Introduced in Definition 3.3 with a scale B fitted to the base model's mean absolute error. No external validation is provided beyond the paper's own comparisons.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What should an AI assessor optimise for?." pith.science (2026). https://pith.science/paper/ZNXBUIQC

@misc{pith2026250200365,
  author       = {Pith},
  title        = {Pith review of: What should an AI assessor optimise for?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZNXBUIQC}},
  note         = {Machine review of arXiv:2502.00365}
}
read the original abstract

An AI assessor is an external, ideally indepen-dent system that predicts an indicator, e.g., a loss value, of another AI system. Assessors can lever-age information from the test results of many other AI systems and have the flexibility of be-ing trained on any loss function or scoring rule: from squared error to toxicity metrics. Here we address the question: is it always optimal to train the assessor for the target metric? Or could it be better to train for a different metric and then map predictions back to the target metric? Us-ing twenty regression and classification problems with tabular data, we experimentally explore this question for, respectively, regression losses and classification scores with monotonic and non-monotonic mappings and find that, contrary to intuition, optimising for more informative met-rics is not generally better. Surprisingly, some monotonic transformations are promising. For example, the logistic loss is useful for minimis-ing absolute or quadratic errors in regression, and the logarithmic score helps maximise quadratic or spherical scores in classification.

Figures

Figures reproduced from arXiv: 2502.00365 by the authors.

Figure 1
Figure 1. For an energy consumption model M1, we want to an￾ticipate the squared error (L + 2 ) for each new example using an ex￾ternal predictor, called assessor. Recommendations to customers are only made when the assessor predicts low L + 2 in the energy consumption estimate. We will explore assessors that optimise for the target loss function (squared loss L + 2 , top) but also assessors that use a proxy loss function (lo… view at source ↗
Figure 2
Figure 2. Software Effort dataset ( [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Histograms of r⊚, the probability estimation for the correct class for some representative datasets, all averaging 255 classification models: CDC Diabetes (most have this shape), JM1 (slightly bimodal) and Higgs (quite bad with values around 0.5). All datasets in Appendix C. datasets from 1. Few cases have probabilities below 0.2, meaning that high confidence misclassifications are very rare. Using the logarithmic s… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: (Left) Score matrix for XGBoost assessor model. (Right) Aggregated Spearman margin matrix for XGBoost assessor model. In both matrices, rows represent target errors and columns proxy errors. Red values indicate poor performance from trying to pre￾dict L◦ by learning L…
Figure 5
Figure 5. Figure 5: Scatter plots for the assessor of the Parkinson’s Disease Rating Scale for RandomForest Regressor base models and assessor model XGBoost. Because the predictions of the assessor tend to the mean, the case where the proxy is signed takes predictions towards 0, and the p…
Figure 6
Figure 6. Figure 6: (Left) Mean score matrix of every possible approach between target and proxy errors. (Right) Aggregated Spearman margin matrix. In both matrices, rows represent target errors and columns proxy errors. Red values indicate poor performance from trying to predict L◦ by l…
Figure 8
Figure 8. Figure 8: (Left) Score matrix for XGBoost assessor model. (Right) Aggregated Spearman margin matrix for XGBoost assessor model. In both matrices, rows represent target scores and columns proxy scores. Red values indicate poor performance from trying to pre￾dict L◦ by learning L…
Figure 10
Figure 10. Figure 10: Functional representation of the three scoring rules we use in this paper, logarithmic (SL), quadratic (SQ) and spherical (SS) scores. A.1.1. DERIVATION OF MONOTONIC TRANSFORMATIONS BETWEEN SCORES We derive mathematical relationships between the scoring rules to under…
Figure 11
Figure 11. Figure 11: Graphical representation of both branches of Eq. 23. The positive branch shows values outside the range of SS. As such, we end up with: SS = p r⊚ 2r⊚ 2 − 2r⊚ + 1 → r⊚ = SS 2 − p SS 2 − SS 4 2SS 2 − 1 (24) And by substitution, in Eq. 15 and Eq. 16, respectively, we obt…
Figure 12
Figure 12. Figure 12: Functional representation of the six losses we use in this paper, signed (L ∓ 1 , L ∓ 2 and L ∓ L ) and unsigned (L + 1 , L + 2 and L + L ). A.2.1. DERIVATION OF MONOTONIC TRANSFORMATIONS BETWEEN LOSSES We now derive mathematical relationships between the loss functio…
Figure 13
Figure 13. Figure 13: visually shows this data generation and splitting approach, illustrating how the original problem features and model characteristics form an example for an assessor. Train Obtain Predict 8 ... 8 9 ... 1 2 3 4 5 6 7 8 9 10 9 10 ... 10 Split ... 8 Original Dataset (TRAI…
Figure 14
Figure 14. Figure 14: Histogram of r⊚. Except for Higgs, the base models are generally good, so there are few high residuals (low r⊚). Abalone Auction Verification BNG EchoMonths California Housing Infrared Thermography Temp. 15 10 5 0 5 10 15 Residuals 0.00 0.05 0.10 0.15 0.20 0.25 Densit…
Figure 15
Figure 15. Figure 15: Histograms of residuals. Some datasets like Auction Verificatoin have some very high residuals, which may contribute to the double penalisation on some losses. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: (Left) Score matrix for (by row) XGBoost, Bayesian ridge, Linear Regression and Feed-forward Neural Network assessor model. (Right) Aggregated Spearman margin matrix for XGBoost assessor model. In both matrices, rows represent target errors and columns proxy errors. R…
Figure 17
Figure 17. Figure 17: Scatter plots for the XGBoost assessor for the Parkinson’s Disease Rating Scale and five base models: XGBRegressor, LGBMRegressor, CatBoostRegressor, RandomForestRegressor and DecisionTreeRegressor. Because the predictions of the assessor tend to the mean, the case wh…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 49 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    and Alimoglu, F

    Alpaydin, E. and Alimoglu, F. Pen-Based Recognition of Handwritten Digits . UCI Machine Learning Repository, 1996. DOI : https://doi.org/10.24432/C5MG6K

    doi:10.24432/c5mg6k 1996

  3. [3]

    and Hoff, A

    Bain, M. and Hoff, A. Chess (King-Rook vs. King) . UCI Repository, 1994

    work page 1994

  4. [4]

    Beaton, A. E. and Tukey, J. W. The fitting of power series, meaning polynomials, illustrated on band-spectroscopic data. Technometrics, 16 0 (2): 0 147--185, 1974

    work page 1974

  5. [5]

    and Kohavi, R

    Becker, B. and Kohavi, R. Adult . UCI Machine Learning Repository, 1996

    work page 1996

  6. [6]

    Robust optimization for deep regression

    Belagiannis, V., Rupprecht, C., Carneiro, G., and Navab, N. Robust optimization for deep regression. In Proc. IEEE ICCV, pp.\ 2830--2838, 2015

    work page 2015

  7. [7]

    Bishop, C. M. and Nasrabadi, N. M. Pattern recognition and machine learning, volume 4. Springer, 2006

    2006

  8. [8]

    MAGIC Gamma Telescope

    Bock, R. MAGIC Gamma Telescope . UCI Machine Learning Repository, 2004

    work page 2004

Show all 68 references
  1. [9]

    A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M

    Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    2021 arXiv

  2. [10]

    Performance metrics (error measures) in machine learning regression, forecasting and prognostics: Properties and typology

    Botchkarev, A. Performance metrics (error measures) in machine learning regression, forecasting and prognostics: Properties and typology. arXiv:1809.03006, 2018

    2018 arXiv

  3. [11]

    A new typology design of performance metrics to measure errors in machine learning regression algorithms

    Botchkarev, A. A new typology design of performance metrics to measure errors in machine learning regression algorithms. Interdisciplinary Journal of Information, Knowledge, and Management, 14: 0 045--076, 2019

    2019

  4. [12]

    Classification and Regression Trees

    Breiman, L., Friedman, J., Stone, C., and Olshen, R. Classification and Regression Trees. Taylor & Francis, 1984. ISBN 9780412048418. URL https://books.google.es/books?id=JwQx-WOmSyQC

    1984

  5. [13]

    Embedding Synthetic Off-Policy Experience for Autonomous Driving via Zero-Shot Curricula

    Bronstein, E., Srinivasan, S., Paul, S., Sinha, A., O'Kelly, M., Nikdel, P., and Whiteson, S. Embedding Synthetic Off-Policy Experience for Autonomous Driving via Zero-Shot Curricula . In 6th CoRL, 2022

    2022

  6. [14]

    Loss functions for binary class probability estimation and classification: Structure and applications

    Buja, A., Stuetzle, W., and Shen, Y. Loss functions for binary class probability estimation and classification: Structure and applications. Working draft, November, 3: 0 13, 2005

    2005

  7. [15]

    D., Martinez-Plumed, F., Tenenbaum, J

    Burnell, R., Schellaert, W., Burden, J., Ullman, T. D., Martinez-Plumed, F., Tenenbaum, J. B., Rutar, D., Cheke, L. G., Sohl-Dickstein, J., Mitchell, M., et al. Rethink reporting of evaluation results in ai. Science, 380 0 (6641): 0 136--138, 2023

    2023

  8. [16]

    Burrows, N. R. Incidence of end-stage renal disease attributed to diabetes among persons with diagnosed diabetes— US and Puerto Rico , 2000--2014. MMWR, 66, 2017

    2000

  9. [17]

    and Lemaire, V

    Candillier, L. and Lemaire, V. Design and analysis of the nomao challenge active learning in the real-world. In Proc. ALRA@ECML-PKDD, pp.\ 1--15, 2012

    2012

  10. [18]

    and Guestrin, C

    Chen, T. and Guestrin, C. Xgboost: A scalable tree boosting system. In Proc. 22nd ACM SIGKDD, KDD ’16. ACM, August 2016. doi:10.1145/2939672.2939785. URL http://dx.doi.org/10.1145/2939672.2939785

    2016

  11. [19]

    J., and Jurman, G

    Chicco, D., Warrens, M. J., and Jurman, G. The coefficient of determination r-squared is more informative than smape, mae, mape, mse and rmse in regression analysis evaluation. Peerj computer science, 7: 0 e623, 2021

    2021

  12. [20]

    Support-vector networks

    Cortes, C. Support-vector networks. Machine Learning, 1995

    1995

  13. [21]

    Learned lessons in credit card fraud detection from a practitioner perspective

    Dal Pozzolo, A., Caelen, O., Le Borgne, Y.-A., Waterschoot, S., and Bontempi, G. Learned lessons in credit card fraud detection from a practitioner perspective. Expert systems with applications, 41 0 (10): 0 4915--4928, 2014

    2014

  14. [22]

    Bold: Dataset and metrics for measuring biases in open-ended language generation

    Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., and Gupta, R. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proc. ACM FAT, pp.\ 862--872, 2021

    2021

  15. [23]

    V., and Awadallah, A

    Ding, D., Mallick, A., Wang, C., Sim, R., Mukherjee, S., Ruhle, V., Lakshmanan, L. V., and Awadallah, A. H. Hybrid llm: Cost-efficient and quality-aware query routing. arXiv:2404.14618, 2024

    2024 arXiv

  16. [24]

    Bootstrap Methods: Another Look at the Jackknife

    Efron, B. Bootstrap Methods: Another Look at the Jackknife . The Annals of Statistics, 7 0 (1): 0 1 -- 26, 1979. doi:10.1214/aos/1176344552. URL https://doi.org/10.1214/aos/1176344552

    1979

  17. [25]

    Estimation and testing of forecast rationality under flexible loss

    Elliott, G., Timmermann, A., and Komunjer, I. Estimation and testing of forecast rationality under flexible loss. The Review of Economic Studies, 72 0 (4): 0 1107--1125, 2005

    2005

  18. [26]

    Reward function design in reinforcement learning

    Eschmann, J. Reward function design in reinforcement learning. Reinforcement learning algorithms: Analysis and Applications, pp.\ 25--33, 2021

    2021

  19. [27]

    Fabra-Boluda, R., Ferri, C., Mart \' nez-Plumed, F., Hern \'a ndez-Orallo, J., and Ram \' rez-Quintana, M. J. Family and prejudice: A behavioural taxonomy of ML techniques. In ECAI 2020, pp.\ 1135--1142. 2020

    2020

  20. [28]

    Unveiling the robustness of machine learning families

    Fabra-Boluda, R., Ferri, C., Ram \' rez-Quintana, M., and Mart \' nez-Plumed, F. Unveiling the robustness of machine learning families. MLST, 5 0 (3): 0 035040, 2024

    2024

  21. [29]

    An introduction to roc analysis

    Fawcett, T. An introduction to roc analysis. Pattern recognition letters, 27 0 (8): 0 861--874, 2006

    2006

  22. [30]

    Regression towards mediocrity in hereditary stature

    Galton, F. Regression towards mediocrity in hereditary stature. The Journal of the Anthropological Institute of Great Britain and Ireland, 15: 0 246--263, 1886. ISSN 09595295, 23972564. URL http://www.jstor.org/stable/2841583

  23. [31]

    and Raftery, A

    Gneiting, T. and Raftery, A. E. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102 0 (477): 0 359--378, 2007

    2007

  24. [32]

    and Ozhegov, E

    Gogolev, S. and Ozhegov, E. Asymmetric loss function in product-level sales forecasting: An empirical comparison. Applied Econometrics, 70: 0 109--121, 2023

    2023

  25. [33]

    A unified view of performance metrics: Translating threshold choice into expected classification loss

    Hern \'a ndez-Orallo, J., Flach, P., and Ferri Ram \' rez, C. A unified view of performance metrics: Translating threshold choice into expected classification loss. Journal of Machine Learning Research, 13: 0 2813--2869, 2012

    2012

  26. [34]

    Training on the test set: Mapping the system-problem space in AI

    Hern \'a ndez-Orallo, J., Schellaert, W., and Mart \' nez-Plumed, F. Training on the test set: Mapping the system-problem space in AI . In AAAI, pp.\ 12256--12261, 2022

    2022

  27. [35]

    Roc curves for regression

    Hernández-Orallo, J. Roc curves for regression. Pattern Recognition, 46 0 (12): 0 3395--3411, 2013. ISSN 0031-3203. doi:https://doi.org/10.1016/j.patcog.2013.06.014. URL https://www.sciencedirect.com/science/article/pii/S0031320313002665

    2013 doi

  28. [36]

    Ho, T. K. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition, volume 1, pp.\ 278--282. IEEE, 1995

    1995

  29. [37]

    J., Bieker, J., Li, X., Jiang, N., Keigwin, B., Ranganath, G., Keutzer, K., and Upadhyay, S

    Hu, Q. J., Bieker, J., Li, X., Jiang, N., Keigwin, B., Ranganath, G., Keutzer, K., and Upadhyay, S. K. Routerbench: A benchmark for multi-llm routing system. arXiv:2403.12031, 2024

    2024 arXiv

  30. [38]

    Huber, P. J. Robust estimation of a location parameter. In Breakthroughs in statistics: Methodology and distribution, pp.\ 492--518. Springer, 1992

    1992

  31. [39]

    and Waegeman, W

    H \"u llermeier, E. and Waegeman, W. Aleatoric and epistemic uncertainty in Machine Learning . Machine learning, 110 0 (3): 0 457--506, 2021

    2021

  32. [40]

    Hyndman, R. J. and Koehler, A. B. Another look at measures of forecast accuracy. International journal of forecasting, 22 0 (4): 0 679--688, 2006

    2006

  33. [41]

    Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Dodds, Z. H., DasSarma, N., Tran-Johnson , E., Johnston, S., El-Showk , S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, ...

    2022 arXiv

  34. [42]

    Song popularity prediction dataset

    Kakkad, Y. Song popularity prediction dataset. GitHub, 2021. URL https://github.com/yashrajkakkad/song-popularity-prediction/blob/master/Dataset/data.csv

    2021

  35. [43]

    Lightgbm: A highly efficient gradient boosting decision tree

    Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. Lightgbm: A highly efficient gradient boosting decision tree. In NIPS, volume 30, 2017

    2017

  36. [44]

    and Barry, R

    Kelley Pace , R. and Barry, R. Sparse spatial autoregressions. Statistics & Probability Letters, 33 0 (3): 0 291--297, 1997. ISSN 0167-7152. doi:https://doi.org/10.1016/S0167-7152(96)00140-X. URL https://www.sciencedirect.com/science/article/pii/S016771529600140X

    1997 doi

  37. [45]

    H., Neumann, F., and Trautmann, H

    Kerschke, P., Hoos, H. H., Neumann, F., and Trautmann, H. Automated Algorithm Selection : Survey and Perspectives . Evolutionary Computation, 27 0 (1): 0 3--45, 2019. ISSN 1063-6560. doi:10.1162/evco_a_00242

    2019 doi

  38. [46]

    Making language models better reasoners with step-aware verifier

    Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., and Chen, W. Making language models better reasoners with step-aware verifier. In Proc. ACL, pp.\ 5315--5333, 2023

    2023

  39. [47]

    Routing to the expert: Efficient reward-guided ensemble of large language models

    Lu, K., Yuan, H., Lin, R., Lin, J., Yuan, Z., Zhou, C., and Zhou, J. Routing to the expert: Efficient reward-guided ensemble of large language models. arXiv preprint arXiv:2311.08692, 2023

    2023 arXiv

  40. [48]

    McCulloch, W. S. and Pitts, W. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5 0 (4): 0 115--133, 1943

    1943

  41. [49]

    and Di Stefano, J

    Menzies, T. and Di Stefano, J. S. How good is your blind spot sampling policy. In IEEE HASE, pp.\ 129--138. IEEE, 2004

    2004

  42. [50]

    A data-driven approach to predict the success of bank telemarketing

    Moro, S., Cortez, P., and Rita, P. A data-driven approach to predict the success of bank telemarketing. Decision Support Systems, 62: 0 22--31, 2014

    2014

  43. [51]

    Abalone, 1995

    Nash, W., Sellers, T., Talbot, S., Cawthorn, A., and Ford, W. Abalone, 1995

    1995

  44. [52]

    Analyzing and predicting verification of data-aware process models–a case study with spectrum auctions

    Ordoni, E., Bach, J., and Fleck, A.-K. Analyzing and predicting verification of data-aware process models–a case study with spectrum auctions. IEEE Access, 10: 0 31699--31713, 2022. doi:10.1109/ACCESS.2022.3154445

    2022

  45. [53]

    Owen, A. B. A robust hybrid of lasso and ridge regression. Contemporary Mathematics, 443 0 (7): 0 59--72, 2007

    2007

  46. [54]

    G., and Hernández-Orallo, J

    Pacchiardi, L., Cheke, L. G., and Hernández-Orallo, J. 100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances. arXiv 2409.03563, 2024

    2024 arXiv

  47. [55]

    Bleu: a method for automatic evaluation of machine translation

    Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. ACL, pp.\ 311--318, 2002

    2002

  48. [56]

    V., and Gulin, A

    Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., and Gulin, A. Catboost: unbiased boosting with categorical features, 2019

    2019

  49. [57]

    D., Le, T

    Romano, J. D., Le, T. T., La Cava, W., Gregg, J. T., Goldberg, D. J., Chakraborty, P., Ray, N. L., Himmelstein, D., Fu, W., and Moore, J. H. Pmlb v1.0: an open source dataset collection for benchmarking machine learning methods. arXiv:2012.00058v2, 2021

    2012 arXiv

  50. [58]

    Rousseeuw, P. J. and Leroy, A. M. Robust regression and outlier detection. John wiley & sons, 2005

    2005

  51. [59]

    A proposal for scaling the scaling laws

    Schellaert, W., Hamon, R., Mart \' nez-Plumed, F., and Hernandez-Orallo, J. A proposal for scaling the scaling laws. In SCALE-LLM, pp.\ 1--8, 2024 a

    2024

  52. [60]

    Analysing the predictability of language model performance

    Schellaert, W., Mart \' nez-Plumed, F., and Hern \'a ndez-Orallo, J. Analysing the predictability of language model performance. ACM TIST, 2024 b

    2024

  53. [61]

    Srivastava, A. et al. Beyond the Imitation Game : Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615, 2022. doi:10.48550/arXiv.2206.04615

    2022 doi arXiv

  54. [62]

    Tipping, M. E. Sparse bayesian learning and the relevance vector machine. JMLR, 1: 0 211–244, 2001. ISSN 1532-4435. doi:10.1162/15324430152748236. URL https://doi.org/10.1162/15324430152748236

    2001 doi

  55. [63]

    A., McSharry, P

    Tsanas, A., Little, M. A., McSharry, P. E., and Ramig, L. O. Accurate telemonitoring of parkinson's disease progression by noninvasive speech tests. IEEE Transactions on Biomedical Engineering, 57: 0 884--893, 2009. URL https://api.semanticscholar.org/CorpusID:7382779

    2009

  56. [64]

    Facial and oral temperature data from a large set of human subject volunteers, 2023

    Wang, Q., Zhou, Y., Ghassemi, P., Chenna, D., Chen, M., Casamento, J., Pfefer, J., and Mcbride, D. Facial and oral temperature data from a large set of human subject volunteers, 2023

    2023

  57. [65]

    Whiteson, D. HIGGS . UCI Machine Learning Repository, 2014

    2014

  58. [66]

    Global health observatory data repository, 2015

    World Health Organization . Global health observatory data repository, 2015

    2015

  59. [67]

    Team formation through an assessor: choosing marl agents in pursuit--evasion games

    Zhao, Y., Ju, L., and Hern \'a ndez-Orallo, J. Team formation through an assessor: choosing marl agents in pursuit--evasion games. Complex & Intelligent Systems, pp.\ 1--20, 2024

    2024

  60. [68]

    Reject before you run: Small assessors anticipate big language models

    Zhou, L., Mart \' nez-Plumed, F., Hern \'a ndez-Orallo, J., Ferri, C., and Schellaert, W. Reject before you run: Small assessors anticipate big language models. In EBeM@ IJCAI, 2022

    2022

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.