Pith. sign in

REVIEW 4 major objections 5 minor 57 references

Bayesian Inference for Correlated Human Experts and Classifiers

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A Bayesian model predicts expert consensus using fewer expert queries.

desk verdict Legitimate extension of INFEXP to identifiable experts; query-savings claim is credible but the evidence needs error bars, ECE at the actual operating point, and an independent baseline. read the letter →

arxiv 2506.05636 v1 pith:B4JUG56V submitted 2025-06-05 cs.LG cs.AI

Markelle Kelly , Alex Boyd , Sam Showalter , Mark Steyvers , Padhraic Smyth This is my paper
classification cs.LGcs.AI MSC 62F1568T0568T37
keywords Bayesianinferenceexpertconsensuspredictionhuman-AIcollaborationactivequeryingmultivariatenormalmodellogisticcalibrationMCMC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a Bayesian generative model over correlated classifier outputs and human expert votes can decide online, per image, which experts to query and when to stop, so that the group's majority label is predicted perfectly with fewer expert queries than existing methods. On two real medical datasets the method reaches zero consensus-prediction error with on average 2.55 versus 3.16 queries (ChestX-ray) and 1.58 versus 1.82 queries (Chaoyang) compared with the strongest baseline. The practical point is that expensive expert time can be saved whenever a cheap pretrained classifier is informative about how a panel of identifiable experts will vote, even when the classifier was trained independently of those experts.

What carries the argument

The load-bearing object is the joint multivariate normal distribution over the additive-logistic-transformed logits of all agents, $z \sim N(\mu, \Sigma)$, with a global temperature $\tau$ that turns latent probability vectors into observed votes. The query policy is expected-entropy reduction of the consensus posterior, evaluated by simulating each candidate expert's vote and re-running the conditional sampling of the unobserved votes. This Gaussian prior is what lets correlations among identifiable experts and the black-box classifier be learned from a small online stream, and the temperature is what recalibrates the vote likelihood so that posterior error estimates match realized error.

What would settle it

Build a dataset whose classifier logits are bimodal, heavy-tailed, or generated from a mixture of two image populations with different expert-classifier correlations, run the method with the paper's hyperparameters and threshold $e=0.01$, and check whether the realized consensus error stays at 0% and whether the expected calibration error stays below 1%: if measured error rises above the threshold or calibration degrades while simpler baselines retain their error levels, the Gaussian assumption is the cause.

Watch

Extended reading notes

Core claim

The central claim is that expert votes can be treated as noisy categorical draws from latent per-expert probability vectors whose logits, together with classifier logits, follow one multivariate normal with a shared mean and covariance, learned online via MCMC. Conditioned on this model, the expected entropy of the consensus distribution after each hypothetical vote identifies the most informative expert to query next, and querying stops when the model's consensus-error estimate falls below a threshold. The paper reports that this procedure consistently achieves zero error on the expert-consensus task across ChestX-ray, Chaoyang, CIFAR-10H, and ImageNet-16H, and does so with fewer queries on average than the INFEXP and confusion-plus-calibration baselines, while maintaining expected calibration error generally below one percent.

Load-bearing premise

The whole argument rests on the assumption that, after the logistic transform, the joint vector of classifier confidences and latent expert confidences is Gaussian with one mean and covariance shared by every image, plus a single vote temperature; if the real joint distribution is non-Gaussian, changes over time, or needs different parameters for different image types, the model's error estimates become miscalibrated and the reported query savings are not guaranteed.

Editorial extensions

If this is right

  • On the medical datasets, the method reaches 0% consensus error with 2.55 queries versus 3.16 (ChestX-ray) and 1.58 versus 1.82 (Chaoyang) compared with INFEXP with epsilon-greedy querying, so expert panels can be consulted less often without losing fidelity to the panel's majority.
  • Because the model's error estimates are calibrated (ECE generally below 1%), the error threshold $e$ is a practical dial: setting $e=0.01$ yields actual error near zero and allows per-example stopping.
  • The same trained model directly supports aggregation functions other than majority vote, such as 'any expert votes positive' or 'all experts agree,' by changing the function $f$, with measurably different query costs.
  • In a distribution-shift experiment on ImageNet-16H, a sliding window of recent examples makes the model increase exploration when accuracy drops, preserving a 0% error rate despite the shift.
  • The framework does not require the classifier to have been trained on expert labels, so it applies when the classifier and experts are mismatched.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the covariance dimension grows as $(K-1)(M+H)$, scaling to many classes or many agents will require low-rank or factored covariance approximations; the paper's Gaussian core would still drive query selection if such structure were imposed.
  • The expected-entropy query rule is a greedy information-gain heuristic; a testable extension would compare it to non-greedy lookahead or to an oracle that queries the expert whose vote changes the consensus most often under the posterior, to see how much of the savings comes from the Gaussian correlation model versus the query rule.
  • The method implicitly assumes the pretrained classifier's predicted probabilities carry signal about the panel's votes; a stress test would swap in a classifier that is calibrated but negatively correlated with experts and measure whether query savings persist.
  • Since the method maintains a posterior over each identifiable expert's latent propensity, it could double as a diagnostic for expert disagreement patterns, flagging images where a single expert is likely to dissent from consensus.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes a Bayesian framework for predicting the consensus labels of a panel of human experts, using classifier probability estimates and partially observed expert votes. The generative model places a multivariate normal distribution on joint classifier and latent human logits, with a global temperature on the vote likelihood. Inference is performed via MCMC, and an expected-entropy criterion selects which expert to query next; querying stops when the model's posterior error estimate falls below a threshold. Experiments on ChestX-ray, Chaoyang, CIFAR-10H, and ImageNet-16H compare the method with two modified baselines and report error-cost curves, ECE values, exploration/exploitation behavior, and a distribution-shift adaptation.

Significance. The paper addresses a practically important and under-explored problem: minimizing expert query cost while predicting expert consensus. The model is general with respect to aggregation functions and identifiable experts, and the authors provide code and data. The empirical ECE results at the tested thresholds and the distribution-shift experiment are useful. However, the central claim of query savings rests on comparisons with same-group modified baselines and on average query counts without variability measures, so the current evidence is suggestive rather than definitive.

major comments (4)
  1. [7.3, Fig. 2] The headline comparison (2.55 vs 3.16 queries for ChestX-ray and 1.58 vs 1.82 for Chaoyang) is presented as averaged over 12 runs, but no standard errors, confidence intervals, or paired tests are reported. The two differences (0.61 and 0.24 queries) are small relative to the likely run-to-run variability of an online procedure on 250-example streams, so without significance information the claim that the method uses fewer queries on average is not statistically supported. Please report per-run values or error bars and a paired significance test for the endpoint query counts.
  2. [7.2, Baselines] Both baselines are modifications of methods from papers with overlapping authorship (Showalter et al. 2024 and Kerrigan et al. 2021). The modifications are non-trivial: INFEXP is augmented with epsilon-greedy expert selection, and the confusion-matrix method is redefined with respect to consensus. No comparison is made to the original methods, to random querying, or to simple uncertainty-sampling policies. This makes it difficult to attribute the observed gains to the Bayesian generative model rather than to baseline implementation choices. At minimum, include simple querying baselines and, if feasible, independent implementations or the original authors' code.
  3. [Section 4, Eq. (3); Algorithm 1] The vote likelihood is not consistently defined. gamma maps probability vectors to logits (Eq. 1), and z_i are defined as logits, but Eq. (3) defines TS(theta_i,tau)=softmax(theta_i/tau) for a probability vector theta_i, whereas Algorithm 1 evaluates TS(gamma(z_i^s),tau), which is undefined as written because gamma(z_i^s) is not defined for a K-1 dimensional logit vector. This ambiguity affects the likelihood in Eq. (4) and the simulated votes in Algorithms 1 and 2. Please define the inverse logistic transformation explicitly and state whether the temperature is applied to logits (softmax(z/tau)) or to probability vectors.
  4. [7.3 and 7.4.1, Table 1] The paper's main operating point is the threshold (or smoothed-curve point) at which 0% error is reached, but the ECE table covers only e in {0.05, 0.025, 0.01}. If the 0%-error query counts are obtained from lowess-smoothed curves or from thresholds outside this range, the reported ECEs do not establish calibration at those operating points. The Gaussian logit assumption in Eq. (2) is strong, and the temperature parameter in Eq. (3) cannot repair conditional misspecification, so calibration at the exact stopping thresholds used for the headline numbers is the load-bearing quantity. Please state which thresholds produce the reported 0%-error points, report raw unsmoothed values, and provide ECE or actual error rates at those same thresholds.
minor comments (5)
  1. [Section 7.3, last paragraph] The sentence 'our method consistently minimizes the error rate (i.e., reaches 0% error)' is ambiguous for CIFAR-10H and ImageNet-16H, where the text states that both baselines never reach 0% error. Please clarify whether the proposed method reaches 0% on all four datasets and, if not, restrict the 0%-error claim to the two medical datasets or rephrase.
  2. [Section 7.3, Fig. 2] If the reported query counts for the 0%-error points are read from lowess-smoothed curves rather than from raw averaged data, this should be stated explicitly; smoothed values can cross zero at query counts where no actual run achieved zero error.
  3. [Section 7.3] The statement that the confusion+calibration baseline never reaches 0% error even when all experts are queried is surprising, since with all votes observed the consensus is known; please clarify the mechanism (e.g., a MAP over a latent consensus model disagreeing with the observed majority) or report the baseline's error when forced to output the observed majority.
  4. [Section 3 and Appendix D] The notation M is used both for the number of classifiers and for the set of classifier indices; please use a different symbol (e.g., a calligraphic M) for the set. Also, Appendix D reports a single set of hyperparameters; since these control the exploration/exploitation trade-off and hence query counts, a sensitivity analysis would help establish robustness of the query-count differences.
  5. [Section 7.4.2, Fig. 4] The distribution-shift experiment reports a 0% error rate after the shift, but no error bars or comparison to a non-adaptive baseline are provided; this would strengthen the claim that the sliding-window adaptation is responsible for the maintained accuracy.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the consensus prediction is a genuine held-out target, and the model's parameters are learned from observed votes rather than fitted to the consensus; the only self-reference is overlapping-author baselines.

full rationale

The paper's derivation chain is self-contained and not circular. The central prediction target is the actual expert consensus (defined as the mode of expert votes, Section 1), and the claimed 0% error results are evaluated against the true expert votes in held-out evaluation sets of 250 examples, not against the model's own posterior estimates. The model parameters (mean mu, covariance Sigma, and temperature tau) are learned from the sequence of observed expert votes and classifier outputs via MCMC (Section 4 and Appendix D), and the posterior predictive in Eq. (4) is a genuine Bayesian computation for unobserved votes conditioned on observed votes; it does not take the consensus as an input. The stopping rule in Section 6 halts when the model's estimated error falls below a threshold, but the paper separately validates calibration via ECE (Table 1), and the headline query counts report actual error, not estimated error. There is no fitted parameter that is renamed as a prediction: the temperature parameter tau is a global calibration parameter, not fitted to the consensus labels, and the reported ECE is compared against actual accuracy. The baselines are modified versions of prior work with overlapping authorship (Showalter et al., 2024 and Kerrigan et al., 2021), which is a comparison-fairness concern rather than a logical circularity; the paper's own method does not rely on those citations for its derivation. No uniqueness theorem, ansatz, or known result is invoked to force the model choice. The multivariate-normal assumption, the logistic-normal transformation, and the LKJ prior are stated as modeling assumptions, not as derived consequences. Overall, the derivation is independent of the target claim, so the circularity burden is very low; the score of 1 reflects only the minor self-citation in the baseline construction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 2 invented entities

The main modeling burden is the multivariate normal assumption over logits, plus a global temperature that corrects calibration. Hyperparameters controlling prior variance and exploration are chosen by hand and influence all reported query counts. The latent logits and temperature are internal constructs with no independent external evidence, and no new physical or ontological entities are proposed.

free parameters (1)
  • Prior hyperparameters sigma_mu, sigma_sigma, sigma_tau, and LKJ concentration eta = 0.1, 1, 0.4, 0.75
    Set by hand in Appendix D and used for all experiments. They control exploration/exploitation and therefore influence the reported query counts and error-cost curves. They are not fitted to held-out data, but other values would change the results, so they are free in the relevant sense.
assumptions (4)
  • domain assumption Joint expert and classifier logits are multivariate normal: z ~ N(mu, Sigma), with shared mean and covariance across examples.
    Section 4, Eq. (2): 'we assume they are all distributed with the same mean mu and covariance Sigma.' This is the core generative premise; if false, predictive distributions for unobserved votes can be biased.
  • domain assumption Expert votes are conditionally independent given latent logits, with a categorical likelihood governed by a temperature-scaled softmax.
    Section 4, Eq. (3) and Algorithm 1. The exact form is inconsistently written (TS applied to a probability vector versus to transformed logits), but conditional independence given z_i is assumed throughout Appendix A.
  • domain assumption All human experts have equal query cost, and classifiers have zero query cost.
    Section 3 states 'we assume that all models have zero cost and that every human expert has the same fixed cost per query.' The stopping rule and cost metric depend on this simplification.
  • domain assumption MCMC with 1,500 warm-up and 2,000 post-warm-up samples per chain yields a faithful approximate posterior.
    Appendix D reports R-hat within +/-0.001 of 1, but the online posterior update uses a finite-sample approximation; any bias here propagates to the querying policy and the calibration results.
invented entities (2)
  • Latent expert logits z_H
    purpose: Couple unobserved expert opinions to classifier probabilities through a shared multivariate normal, enabling prediction of unobserved votes.
    Standard latent-variable modeling construct; no external falsifiable handle, only internal to the model.
  • Global temperature tau
    purpose: Correct miscalibration of the predictive distribution over human votes.
    Introduced in Section 4 as a correction after noting predictive distributions 'have the potential to be miscalibrated'; inferred from data, with no independent evidence outside the model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bayesian Inference for Correlated Human Experts and Classifiers." pith.science (2026). https://pith.science/paper/B4JUG56V

@misc{pith2026250605636,
  author       = {Pith},
  title        = {Pith review of: Bayesian Inference for Correlated Human Experts and Classifiers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B4JUG56V}},
  note         = {Machine review of arXiv:2506.05636}
}
read the original abstract

Applications of machine learning often involve making predictions based on both model outputs and the opinions of human experts. In this context, we investigate the problem of querying experts for class label predictions, using as few human queries as possible, and leveraging the class probability estimates of pre-trained classifiers. We develop a general Bayesian framework for this problem, modeling expert correlation via a joint latent representation, enabling simulation-based inference about the utility of additional expert queries, as well as inference of posterior distributions over unobserved expert labels. We apply our approach to two real-world medical classification problems, as well as to CIFAR-10H and ImageNet-16H, demonstrating substantial reductions relative to baselines in the cost of querying human experts while maintaining high prediction accuracy.

Figures

Figures reproduced from arXiv: 2506.05636 by the authors.

Figure 1
Figure 1. Graphical representation of the assumed generative model. Grey symbols are observed and assumed known, white are random and unobserved. Circles are random variables and rectan￾gles are deterministic transformations of other values. Notation for hyperparameters and time-dependence t is omitted for simplicity. ∆K, i = 1, ..., M. 2 Thus, we can collect predictions from H + M total agents (we will use “agents” to refer … view at source ↗
Figure 2
Figure 2. Average number of expert queries versus classification error rate for each of our four datasets. for each example. For any specific threshold value, our model sequentially processes each example x (t) . For each x (t) , the model decides whether to query an expert (and, if so, which expert to query), makes that query, and repeats the process, stopping when the estimated error is below the threshold e. At that point,… view at source ↗
Figure 3
Figure 3. The error threshold vs. actual error rate of our model. each set. The contrast between the two demonstrates this explore/exploit trade-off: more querying in the first 50 exam￾ples as the model learns parameters (and makes predictions), and less querying in the last 50 as it focuses on prediction. X-Ray Chaoyang CIFAR ImageNet First 50 2.56 2.10 1.98 2.07 Last 50 1.57 1.61 1.10 1.32 [PITH_FULL_IMAGE:figures/full_fig… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The average number of queries issued by our model over time (averaged over 48 experimental runs) for a modified version of ImageNet-16H with a distribution shift at t=125. error rate despite the experts’ drops in accuracy. 7.4.3. ALTERNATIVE AGGREGATION FUNCTIONS Altho…
Figure 5
Figure 5. Figure 5: Error vs. cost for the ChestX-ray dataset in predicting majority vote (green), in comparison to predicting whether any expert will vote 1 (red) and whether the vote will be unanimous (purple). Error-cost curves for the ChestX-ray dataset, using these aggregation functi…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 51 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Xu, W., Singh, A., Osborne, M

    Adachi, M., Chau, S. L., Xu, W., Singh, A., Osborne, M. A., and Muandet, K. Bayesian optimization for building social-influence-free consensus. arXiv preprint arXiv:2502.07166, 2025

    arXiv 2025

  3. [3]

    and Shen, S

    Atchison, J. and Shen, S. M. Logistic-normal distributions: Some properties and uses. Biometrika, 67 0 (2): 0 261--272, 1980

    1980

  4. [4]

    Bansal, G., Nushi, B., Kamar, E., Horvitz, E., and Weld, D. S. Is the most accurate AI the best teammate? optimizing AI for teamwork. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 11405--11414, 2021 a

    work page 2021

  5. [5]

    T., and Weld, D

    Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., and Weld, D. Does the whole exceed its parts? the effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21, New York, NY, USA, 2021 b . Association for Computing Machinery

    work page 2021

  6. [6]

    R., Scarlata, C., Fortson, L

    Beck, M. R., Scarlata, C., Fortson, L. F., Lintott, C. J., Simmons, B., Galloway, M. A., Willett, K. W., Dickinson, H., Masters, K. L., Marshall, P. J., et al. Integrating human and machine intelligence in galaxy morphology classification tasks. Monthly Notices of the Royal Astronomical Society, 476 0 (4): 0 5516--5534, 2018

    work page 2018

  7. [7]

    M., Kamalaruban, P., Kallina, E., Weller, A., and Talwalkar, A

    Bhatt, U., Chen, V., Collins, K. M., Kamalaruban, P., Kallina, E., Weller, A., and Talwalkar, A. Learning personalized decision support policies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp.\ 14203--14211, 2025

    work page 2025

  8. [8]

    L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B

    Bien, N., Rajpurkar, P., Ball, R. L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B. N., Yeom, K. W., Shpanskaya, K., Halabi, S., Zucker, E., Fanton, G., Amanatullah, D. F., Beaulieu, C. F., Riley, G. M., Stewart, R. J., Blankenberg, F. G., Larson, D. B., Jones, R. H., Langlotz, C. P., Ng, A. Y., and Lungren, M. P. Deep-learning-assisted diagnosis...

    work page 2018

Show all 57 references
  1. [9]

    and Lafferty, J

    Blei, D. and Lafferty, J. Correlated topic models. In Advances in Neural Information Processing Systems, volume 18, pp.\ 147--154. MIT Press, 2006

    2006

  2. [10]

    R., and Puranam, P

    Choudhary, V., Marchetti, A., Shrestha, Y. R., and Puranam, P. Human- AI ensembles: When can they work? Journal of Management, pp.\ 01492063231194968, 2023

    2023

  3. [11]

    and Rodriguez, M

    Corvelo Benz, N. and Rodriguez, M. Human-aligned calibration for AI -assisted decision making. In Advances in Neural Information Processing Systems, volume 36, pp.\ 14609--14636, 2024

    2024

  4. [12]

    D., O'Doherty, J

    Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., and Dolan, R. J. Cortical substrates for exploratory decisions in humans. Nature, 441 0 (7095): 0 876--879, 2006

    2006

  5. [13]

    A case for humans-in-the-loop: Decisions in the presence of erroneous algorithmic scores

    De-Arteaga, M., Fogliato, R., and Chouldechova, A. A case for humans-in-the-loop: Decisions in the presence of erroneous algorithmic scores. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20. ACM, April 2020

    2020

  6. [14]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops), pp.\ 248--255, Los Alamitos, CA, USA, jun 2009. ...

    2009

  7. [15]

    Human-algorithm collaboration: Achieving complementarity and avoiding unfairness

    Donahue, K., Chouldechova, A., and Kenthapadi, K. Human-algorithm collaboration: Achieving complementarity and avoiding unfairness. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT '22, pp.\ 1639–1656, New York, NY, USA, 2022. ACM Press

    2022

  8. [16]

    A cognitive framework for delegation between error-prone AI and human agents

    Fuchs, A., Passarella, A., and Conti, M. A cognitive framework for delegation between error-prone AI and human agents. In 2022 IEEE International Conference on Smart Computing (SMARTCOMP), pp.\ 317--322, 2022

    2022

  9. [17]

    Exploring user heterogeneity in human delegation behavior towards AI

    F \"u gener, A., Grahl, J., Gupta, A., Ketter, W., and Taudien, A. Exploring user heterogeneity in human delegation behavior towards AI . In International Conference on Information Systems (ICIS), 2021

    2021

  10. [18]

    K., Segal, A., Kamar, E., Horvitz, E., Lintott, C., and Walmsley, M

    Gal, Y. K., Segal, A., Kamar, E., Horvitz, E., Lintott, C., and Walmsley, M. A new workflow for human- AI collaboration in citizen science. In Proceedings of the 2022 ACM Conference on Information Technology for Social Good, GoodIT '22, pp.\ 89–95, New York, NY, USA, 2022. ACM Press

    2022

  11. [19]

    K., and Lease, M

    Gao, R., Saar-Tsechansky, M., De-Arteaga, M., Han, L., Sun, W., Lee, M. K., and Lease, M. Learning complementary policies for human- AI teams. arXiv preprint arXiv:2302.02944, 2023

    2023

  12. [20]

    and Chen, Y

    Green, B. and Chen, Y. The principles and limits of algorithm-in-the-loop decision making. In Proc. ACM Hum.-Comput. Interact., volume 3, New York, NY, USA, 2019. ACM Press

    2019

  13. [21]

    Forming effective human- AI teams: Building machine learning models that complement the capabilities of multiple experts

    Hemmer, P., Schellhammer, S., Vossing, M., Jakubik, J., and Satzger, G. Forming effective human- AI teams: Building machine learning models that complement the capabilities of multiple experts. In International Joint Conference on Artificial Intelligence, pp.\ 2478--2484, 2022

    2022

  14. [22]

    o ssing, M., Jakubik, J., and K \

    Hemmer, P., Thede, L., V \"o ssing, M., Jakubik, J., and K \"u hl, N. Learning to defer with limited expert predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 6002--6011, 2023

    2023

  15. [23]

    Online decision mediation

    Jarrett, D., H \"u y \"u k, A., and van der Schaar, M. Online decision mediation. Advances in Neural Information Processing Systems, 35: 0 1790--1805, 2022

    2022

  16. [24]

    Jouini, M. N. and Clemen, R. T. Copula models for aggregating expert opinions. Operations Research, 44 0 (3): 0 444--457, 1996

    1996

  17. [25]

    Combining human predictions with model probabilities via confusion matrices and calibration

    Kerrigan, G., Smyth, P., and Steyvers, M. Combining human predictions with model probabilities via confusion matrices and calibration. Advances in Neural Information Processing Systems, 34: 0 4421--4434, 2021

    2021

  18. [26]

    Towards unbiased and accurate deferral to multiple experts

    Keswani, V., Lease, M., and Kenthapadi, K. Towards unbiased and accurate deferral to multiple experts. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp.\ 154--165, 2021

    2021

  19. [27]

    and Ghahramani, Z

    Kim, H.-C. and Ghahramani, Z. Bayesian classifier combination. In Lawrence, N. D. and Girolami, M. (eds.), Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pp.\ 619--627, La ...

    2012

  20. [28]

    and Hinton, G

    Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Technical report, University of Toronto, ON, Canada, 2009

    2009

  21. [29]

    S., and Ma, T

    Kumar, A., Liang, P. S., and Ma, T. Verified uncertainty calibration. In Advances in Neural Information Processing Systems, pp.\ 3787--3798, 2019

    2019

  22. [30]

    Generating random correlation matrices based on vines and extended onion method

    Lewandowski, D., Kurowicka, D., and Joe, H. Generating random correlation matrices based on vines and extended onion method. Journal of Multivariate Analysis, 100 0 (9): 0 1989--2001, 2009

    1989

  23. [31]

    Lindley, D. V. Making Decisions. Wiley, 1986

    1986

  24. [32]

    Understanding the effect of out-of-distribution examples and interactive explanations on human- AI decision making

    Liu, H., Lai, V., and Tan, C. Understanding the effect of out-of-distribution examples and interactive explanations on human- AI decision making. Proc. ACM Hum.-Comput. Interact., 5 0 (CSCW2), 2021

    2021

  25. [33]

    Who should I trust: AI or myself? leveraging human and AI correctness likelihood to promote appropriate trust in AI -assisted decision-making

    Ma, S., Lei, Y., Wang, X., Zheng, C., Shi, C., Yin, M., and Ma, X. Who should I trust: AI or myself? leveraging human and AI correctness likelihood to promote appropriate trust in AI -assisted decision-making. In Proceedings of the 2023 CHI Conference on Human Factors in Compu...

    2023

  26. [34]

    Predict responsibly: Improving fairness and accuracy by learning to defer

    Madras, D., Pitassi, T., and Zemel, R. Predict responsibly: Improving fairness and accuracy by learning to defer. In Advances in Neural Information Processing Systems, volume 31, pp.\ 6150--6160, 2018

    2018

  27. [35]

    Who should predict? exact algorithms for learning to defer to humans

    Mozannar, H., Lang, H., Wei, D., Sattigeri, P., Das, S., and Sontag, D. Who should predict? exact algorithms for learning to defer to humans. In International Conference on AI and Statistics , pp.\ 10520--10545. PMLR, 2023

    2023

  28. [36]

    P., Ye, W., Yang, J., Pilgrim, R., Kazemzadeh, S., et al

    Nabulsi, Z., Sellergren, A., Jamshy, S., Lau, C., Santos, E., Kiraly, A. P., Ye, W., Yang, J., Pilgrim, R., Kazemzadeh, S., et al. Deep learning for distinguishing normal versus abnormal chest radiographs and generalization to two unseen diseases tuberculosis and COVID -19. Sc...

    2021

  29. [37]

    and Chen, Y

    Noti, G. and Chen, Y. Learning when to advise human decision makers. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI '23, 2023

    2023

  30. [38]

    and Grofman, B

    Owen, G. and Grofman, B. Information Pooling and Group Decision Making. JAI Press, 1986

    1986

  31. [39]

    C., Battleday, R

    Peterson, J. C., Battleday, R. M., Griffiths, T. L., and Russakovsky, O. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 9617--9626, 2019

    2019

  32. [40]

    and S trumbelj, E

    Pir s , G. and S trumbelj, E. Bayesian combination of probabilistic classifiers using multivariate normal mixtures. Journal of Machine Learning Research, 20 0 (51): 0 1--18, 2019

    2019

  33. [41]

    and Yee, M

    Raman, N. and Yee, M. Improving learning-to-defer algorithms through fine-tuning. arXiv:2112.10768, 2021

    2021 arXiv

  34. [42]

    A taxonomy of human and ML strengths in decision-making to investigate human- ML complementarity

    Rastogi, C., Leqi, L., Holstein, K., and Heidari, H. A taxonomy of human and ML strengths in decision-making to investigate human- ML complementarity. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 11, pp.\ 127--139, 2023

    2023

  35. [43]

    Appropriate reliance on AI advice: Conceptualization and the effect of explanations

    Schemmer, M., Kuehl, N., Benz, C., Bartos, A., and Satzger, G. Appropriate reliance on AI advice: Conceptualization and the effect of explanations. In Proceedings of the 28th International Conference on Intelligent User Interfaces, pp.\ 410--422, 2023

    2023

  36. [44]

    J., Smyth, P., and Steyvers, M

    Showalter, S., Boyd, A. J., Smyth, P., and Steyvers, M. Bayesian online learning for consensus prediction. In International Conference on Artificial Intelligence and Statistics, pp.\ 2539--2547. PMLR, 2024

    2024

  37. [45]

    Bayesian modeling of human– AI complementarity

    Steyvers, M., Tejeda, H., Kerrigan, G., and Smyth, P. Bayesian modeling of human– AI complementarity. Proceedings of the National Academy of Sciences, 119 0 (11): 0 e2111547119, 2022

    2022

  38. [46]

    Straitouri, E., Wang, L., Okati, N., and Rodriguez, M. G. Improving expert predictions with conformal prediction. In International Conference on Machine Learning, pp.\ 32633--32653. PMLR, 2023

    2023

  39. [47]

    The Wisdom of Crowds

    Surowiecki, J. The Wisdom of Crowds. Vintage, 2005

    2005

  40. [48]

    A., Brandon, C

    Tang, Y.-X., Tang, Y.-B., Peng, Y., Yan, K., Bagheri, M., Redd, B. A., Brandon, C. J., Lu, Z., Han, M., Xiao, J., et al. Automated abnormality classification of chest radiographs using deep convolutional neural networks. NPJ Digital Medicine, 3 0 (1): 0 70, 2020

    2020

  41. [49]

    and Rothkopf, C

    Trick, S. and Rothkopf, C. Bayesian classifier fusion with an explicit model of correlation. In Camps-Valls, G., Ruiz, F. J. R., and Valera, I. (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Mach...

  42. [50]

    P., and Ferrari, V

    Tudor Ionescu, R., Alexe, B., Leordeanu, M., Popescu, M., Papadopoulos, D. P., and Ferrari, V. How hard can it be? estimating the difficulty of visual search in an image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 2157--2166, 2016

    2016

  43. [51]

    and Nalisnick, E

    Verma, R. and Nalisnick, E. Calibrated learning to defer with one-vs-all classifiers. In International Conference on Machine Learning, pp.\ 22184--22202. PMLR, 2022

    2022

  44. [52]

    Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles

    Verma, R., Barrej \'o n, D., and Nalisnick, E. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. In International Conference on Artificial Intelligence and Statistics, pp.\ 11415--11434. PMLR, 2023

    2023

  45. [53]

    Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., and Summers, R. M. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE Conference on Computer Vision and Patt...

    2017

  46. [54]

    Wilson, K. J. An investigation of dependence in expert judgement studies with multiple experts. International Journal of Forecasting, 33 0 (1): 0 325--336, 2017

    2017

  47. [55]

    Winkler, R. L. Combining probability distributions from dependent information sources. Management Science, 27 0 (4): 0 479--488, 1981

    1981

  48. [56]

    V., and Bellamy, R

    Zhang, Y., Liao, Q. V., and Bellamy, R. K. E. Effect of confidence and explanation on accuracy and trust calibration in AI -assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* ’20. ACM, January 2020

    2020

  49. [57]

    Hard sample aware noise robust learning for histopathology image classification

    Zhu, C., Chen, W., Peng, T., Wang, Y., and Jin, M. Hard sample aware noise robust learning for histopathology image classification. IEEE transactions on Medical Imaging, 41 0 (4): 0 881--894, 2021

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.