arxiv: 2605.08388 · v1 · submitted 2026-05-08 · 💻 cs.AI

Recognition: no theorem link

PLACO: A Multi-Stage Framework for Cost-Effective Performance in Human-AI Teams

Pranavkumar Mallela , Vinay Kumar , Shashi Shekhar Jha , Shweta Jain

Authors on Pith no claims yet

Pith reviewed 2026-05-12 01:42 UTC · model grok-4.3

classification 💻 cs.AI

keywords Human-AI collaborationclassificationBayesian combinationcost-effective labelingmulti-stage frameworklabel fusion

0 comments

The pith

PLACO adds staging to Bayesian human-AI label combination so teams reach target accuracy with fewer human queries.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces PLACO, a multi-stage framework that extends an existing Bayesian method for merging a deterministic human label with a probabilistic model output. The extension lets the system decide at each stage whether to request the human label or accept the model's prediction, balancing accuracy against the cost of human effort. A sympathetic reader would care because many real classification pipelines still rely on expensive or slow human input, and a staged approach promises to cut that cost without sacrificing performance. The work keeps the core conditional-independence assumption from the prior Bayesian combiner but adds decision logic to apply it selectively.

Core claim

PLACO is a multi-stage framework for cost-effective performance in Human-AI classification teams. It builds directly on the Bayesian combination rule that fuses a deterministic human labeler with a probabilistic classifier under the assumption of conditional independence given the ground truth, using instance-level model probabilities and class-level human calibration. The multi-stage structure adds sequential decision points that determine whether to query the human or rely on the current model output, thereby reducing the expected number of human interventions while preserving or improving final accuracy.

What carries the argument

A multi-stage decision process that sequentially applies the Bayesian combiner and routes the instance to the human only when the current posterior uncertainty exceeds a cost-adjusted threshold.

If this is right

Fewer human labels are needed to reach any chosen accuracy level compared with always querying the human.
The framework inherits calibrated probabilities from the base Bayesian combiner, so downstream decisions remain probabilistically coherent.
The approach applies to any task where a deterministic human labeler and a probabilistic model are available.
Expected cost scales with the fraction of instances routed to the human at later stages.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same staged routing logic could be tested on tasks with multiple humans or multiple models to further reduce per-instance cost.
If the independence assumption weakens in practice, the multi-stage version may accumulate more error than a single-stage version, suggesting a natural diagnostic experiment.
Deployment in production would require estimating the per-stage query cost in advance so the threshold can be set without hindsight.

Load-bearing premise

Human and model outputs remain conditionally independent given the true label, so the Bayesian update stays valid at every stage.

What would settle it

On a labeled dataset, run the single-stage Bayesian combiner and the PLACO multi-stage version with the same accuracy target; if the multi-stage version does not reduce average human queries while matching accuracy, the cost-effectiveness claim fails.

Figures

Figures reproduced from arXiv: 2605.08388 by Pranavkumar Mallela, Shashi Shekhar Jha, Shweta Jain, Vinay Kumar.

**Figure 1.** Figure 1: Illustration of Probabilistic Labeller-Assisted Cost Optimization (PLACO) Framework for Human-AI Collaboration a single label from a human’s non-probabilistic output and an independently trained model’s probabilistic output. Further enhancing this idea, Singh et al. [21] combine the non-probabilistic output from a subset of humans. They also show that the increase in accuracy is non-monotone, thus accentu… view at source ↗

**Figure 2.** Figure 2: Estimation Match Comparison on CIFAR-10H and ImageNet-16H respectively across different human configurations. Estimation match is the average fraction of correctly estimated human labels on a given instance. A CNN model with accuracy 56% is used as AI model for probabilistic output for CIFAR-10H and a another CNN model with accuracy 43% is used for ImageNet-16H to combine with human labels. 4.2 Value Funct… view at source ↗

**Figure 3.** Figure 3: Learning curve with different subset selection methods average over 10 runs. Each plot corresponds to a different human configuration of varying accuracies. Specifically, the number of humans in the configurations are 5, 7, 10 and 15 (from left to right in each row), with accuracies ranging from 0.3 to 0.9. The first row corresponds to CIFAR-10H and the second corresponds to ImageNet-16H. A CNN model with … view at source ↗

**Figure 4.** Figure 4: Accuracy vs Cost Trade-Off Scatter Plot for with different subset selection methods average over 10 runs. Each plot corresponds to a different human configuration of varying accuracies. Specifically, the number of humans in the configurations are 5, 7, 10 and 15 (from left to right in each row), with accuracies ranging from 0.3 to 0.9. The first row corresponds to CIFAR-10H (5000 training instances) and th… view at source ↗

read the original abstract

Human-AI teams play a pivotal role in improving overall system performance when neither the human nor the model can achieve such performance on their own. With the advent of powerful and accessible Generative AI models, several mundane tasks have morphed into Human-AI team tasks. From writing essays to developing advanced algorithms, humans have found that using AI assistance has led to an accelerated work pace like never before. In classification tasks, where the final output is a single hard label, it is crucial to address the combination of human and model output. Prior work elegantly solves this problem using Bayes rule, using the assumption that human and model output are conditionally independent given the ground truth. Specifically, it discusses a combination method to combine a single deterministic labeler (the human) and a probabilistic labeler (the classifier model) using the model's instance-level and the human's class-level calibrated probabilities.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

PLACO layers cost stages onto an existing Bayesian human-model combiner, but the whole thing still rests on an untested conditional independence assumption.

read the letter

The main thing to know is that PLACO takes the prior Bayesian method for combining a human labeler with a probabilistic classifier and adds multi-stage logic to control when and how much human input is used, with the goal of cutting costs while holding accuracy steady. That framing is the actual addition here. The paper does a reasonable job laying out why cost matters in human-AI classification tasks once generative models are in the loop, and it keeps the reference to the earlier Bayes-rule combination clear. The multi-stage structure at least gives a concrete way to think about staged decisions rather than a single all-or-nothing fusion step. That part is useful for anyone who has to budget human time in deployed systems. The soft spot is exactly what the stress test flags: the combination step inherits the conditional-independence assumption between human and model outputs given the ground truth, and nothing in the description shows a check, relaxation, or sensitivity test for it. If errors correlate on the same hard cases, the downstream cost calculations lose their grounding. The abstract also gives no equations, no algorithm pseudocode, and no experimental numbers, so it is impossible to tell whether the claimed savings are real or just fitted parameters around the old combiner. The citation pattern is fine because it points back to the prior work, but that does not substitute for new validation. This paper is mainly for researchers already working on human-AI collaboration who want ideas for cost-aware pipelines. A reader in that niche could pull the staged-decision concept and try it themselves, but anyone outside that narrow area will not get much. It is worth sending to peer review so the authors can supply the missing experiments and address whether the independence assumption holds in their data.

Referee Report

2 major / 1 minor

Summary. The paper introduces PLACO, a multi-stage framework for cost-effective performance in Human-AI teams on classification tasks. It extends prior Bayesian combination of a deterministic human labeler and a probabilistic classifier (via Bayes rule under conditional independence given ground truth) by adding cost-aware stages that leverage instance-level model probabilities and class-level human calibration.

Significance. If the multi-stage extension is shown to preserve the benefits of the Bayesian combiner while demonstrably reducing cost without degrading accuracy, the work could provide a practical template for deploying Human-AI systems on mundane labeling tasks. The explicit incorporation of cost parameters into the staged decision process is a potentially useful engineering contribution, provided the inherited independence assumption holds or is relaxed.

major comments (2)

[Abstract / combination step] Abstract and combination description: the cost-effectiveness claims rest on the Bayesian posterior derived from the conditional-independence assumption between human and model outputs given ground truth. No derivation, sensitivity analysis, or empirical check of this assumption appears in the manuscript; if human and model errors correlate through shared task features or generative-AI artifacts, the claimed performance-cost gains do not necessarily follow from the cited prior work.
[Framework description] Multi-stage framework: the manuscript supplies no equations, algorithm pseudocode, or experimental protocol showing how the additional stages integrate with the Bayesian combiner or how cost parameters are estimated without introducing new free parameters that undermine the 'cost-effective' claim.

minor comments (1)

[Abstract] The abstract would benefit from a concise statement of the precise novelty of the multi-stage design relative to the single-stage Bayesian baseline.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major comment below and describe the revisions we will make to strengthen the manuscript.

read point-by-point responses

Referee: [Abstract / combination step] Abstract and combination description: the cost-effectiveness claims rest on the Bayesian posterior derived from the conditional-independence assumption between human and model outputs given ground truth. No derivation, sensitivity analysis, or empirical check of this assumption appears in the manuscript; if human and model errors correlate through shared task features or generative-AI artifacts, the claimed performance-cost gains do not necessarily follow from the cited prior work.

Authors: The conditional independence assumption is inherited directly from the prior Bayesian combination work cited in the manuscript, where the derivation via Bayes' rule is already established. We did not repeat the derivation or add new checks in the current version. To address the concern regarding possible error correlations, we will include a dedicated sensitivity analysis section in the revision. This will provide both theoretical discussion of robustness under mild dependence and empirical results on the experimental datasets to confirm that the reported performance-cost benefits hold. revision: yes
Referee: [Framework description] Multi-stage framework: the manuscript supplies no equations, algorithm pseudocode, or experimental protocol showing how the additional stages integrate with the Bayesian combiner or how cost parameters are estimated without introducing new free parameters that undermine the 'cost-effective' claim.

Authors: We agree that the multi-stage integration and cost-parameter handling require more explicit presentation. The stages leverage instance-level model probabilities and class-level human calibration to decide deferral or direct use of the Bayesian posterior, with costs drawn from pre-calibrated quantities. In the revised manuscript we will add the complete set of decision equations, a pseudocode algorithm, and a clear protocol for cost estimation that introduces no additional free parameters beyond those already present in the base model and calibrations. revision: yes

Circularity Check

0 steps flagged

No significant circularity; multi-stage extension builds on externally cited Bayesian combination without reducing to self-inputs.

full rationale

The paper's core contribution is a multi-stage framework extending a prior Bayesian combination method for human-AI label fusion. The abstract explicitly attributes the combination rule and conditional-independence assumption to 'prior work' without claiming to derive or prove it internally. No equations, fitted parameters, or predictions in the provided text reduce by construction to the inputs (e.g., no self-definitional re-use of the independence assumption as a 'prediction,' no renaming of known results, and no load-bearing self-citation chain where the central claim collapses to unverified prior work by the same authors). The new stages for cost-effectiveness introduce independent structure around the cited base method. Per the rules, an inherited assumption from external prior work is not circularity when the paper does not present it as newly derived or force the result by definition. This is the common honest non-finding for papers that properly cite foundations.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The framework rests on the conditional-independence assumption from prior work and on unspecified cost models for the new stages; no free parameters or invented entities are named in the abstract.

axioms (1)

domain assumption Human and model outputs are conditionally independent given the ground truth
Explicitly referenced in the abstract as the assumption used by prior work that the paper builds upon.

pith-pipeline@v0.9.0 · 5463 in / 1136 out tokens · 42014 ms · 2026-05-12T01:42:33.091344+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages

[1]

R. W. Andrews, J. M. Lilly, D. Srivastava, and K. M. Feigh. The role of shared mental models in human-ai teams: a theoretical review.The- oretical Issues in Ergonomics Science, 24(2):129–175, 2023

work page 2023
[2]

arXiv preprint arXiv:2205.01411 , booktitle =

V . Babbar, U. Bhatt, and A. Weller. On the utility of prediction sets in human-ai teams.arXiv preprint arXiv:2205.01411, 2022

work page arXiv 2022
[3]

Bansal, B

G. Bansal, B. Nushi, E. Kamar, E. Horvitz, and D. S. Weld. Is the most accurate AI the best teammate? optimizing AI for teamwork. InThirty- Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty- Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Arti- ficial In...

work page doi:10.1609/aaai.v35i13.17359 2021
[4]

Bondi, R

E. Bondi, R. Koster, H. Sheahan, M. Chadwick, Y . Bachrach, T. Cemgil, U. Paquet, and K. Dvijotham. Role of human-ai interaction in selective prediction. InProceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pages 5286–5294, 2022

work page 2022
[5]

O. Caelen. A bayesian interpretation of the confusion matrix.Annals of Mathematics and Artificial Intelligence, 81(3-4):429–450, 2017

work page 2017
[6]

Fuchs, A

A. Fuchs, A. Passarella, and M. Conti. Optimizing risk-averse human-ai hybrid teams.arXiv preprint arXiv:2403.08386, 2024

work page arXiv 2024
[7]

R. Gao, M. Saar-Tsechansky, M. De-Arteaga, L. Han, M. K. Lee, and M. Lease. Human-ai collaboration with bandit feedback. In Z.-H. Zhou, editor,Proceedings of the Thirtieth International Joint Confer- ence on Artificial Intelligence, IJCAI-21, pages 1722–1728. Interna- tional Joint Conferences on Artificial Intelligence Organization, 8 2021. Main Track

work page 2021
[8]

Gupta, S

S. Gupta, S. Jain, S. S. Jha, P.-A. Hsiung, and M.-H. Wang. Take expert advice judiciously: Combining groupwise calibrated model probabili- ties with expert predictions. InECAI 2023, pages 956–963. IOS Press, 2023

work page 2023
[9]

Hemmer, S

P. Hemmer, S. Schellhammer, M. Vössing, J. Jakubik, and G. Satzger. Forming effective human-ai teams: building machine learning models that complement the capabilities of multiple experts.arXiv preprint arXiv:2206.07948, 2022

work page arXiv 2022
[10]

Hemmer, L

P. Hemmer, L. Thede, M. Vössing, J. Jakubik, and N. Kühl. Learning to defer with limited expert predictions. InProceedings of the AAAI Con- ference on Artificial Intelligence, volume 37, pages 6002–6011, 2023

work page 2023
[11]

Hendrycks and K

D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenRe- view.net, 2017. URL https://openreview.net/forum?id=Hkg4TI9xl

work page 2017
[12]

S. Jain, S. Gujar, S. Bhat, O. Zoeter, and Y . Narahari. A quality as- suring, cost optimal multi-armed bandit mechanism for expertsourcing. Artificial Intelligence, 254:44–63, 2018

work page 2018
[13]

Kerrigan, P

G. Kerrigan, P. Smyth, and M. Steyvers. Combining human predic- tions with model probabilities via confusion matrices and calibration. Advances in Neural Information Processing Systems, 34:4421–4434, 2021

work page 2021
[14]

Keswani, M

V . Keswani, M. Lease, and K. Kenthapadi. Towards unbiased and accu- rate deferral to multiple experts. InProceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 154–165, 2021

work page 2021
[15]

Lamberson and S

P. Lamberson and S. E. Page. Optimal forecasting groups.Management Science, 58(4):805–810, 2012

work page 2012
[16]

Leitão, P

D. Leitão, P. Saleiro, M. A. T. Figueiredo, and P. Bizarro. Human- ai collaboration in decision-making: Beyond learning to defer.CoRR, abs/2206.13202, 2022. doi: 10.48550/arXiv.2206.13202. URL https: //doi.org/10.48550/arXiv.2206.13202

work page doi:10.48550/arxiv.2206.13202 2022
[17]

Madras, T

D. Madras, T. Pitassi, and R. Zemel. Predict responsibly: improving fairness and accuracy by learning to defer.Advances in Neural Infor- mation Processing Systems, 31, 2018

work page 2018
[18]

Martinez, K

J. Martinez, K. Gal, E. Kamar, and L. H. Lelis. Improving the performance-compatibility tradeoff with personalized objective func- tions. InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 5967–5974, 2021

work page 2021
[19]

Mosqueira-Rey, E

E. Mosqueira-Rey, E. Hernández-Pereira, D. Alonso-Ríos, J. Bobes- Bascarán, and Á. Fernández-Leal. Human-in-the-loop machine learn- ing: a state of the art.Artificial Intelligence Review, 56(4):3005–3054, 2023

work page 2023
[20]

Mozannar and D

H. Mozannar and D. Sontag. Consistent estimators for learning to defer to an expert. InInternational Conference on Machine Learning, pages 7076–7087. PMLR, 2020

work page 2020
[21]

Singh, S

S. Singh, S. Jain, and S. S. Jha. On subset selection of multiple humans to improve human-ai team accuracy. InProceedings of the 2023 Inter- national Conference on Autonomous Agents and Multiagent Systems, pages 317–325, 2023

work page 2023
[22]

Steyvers and H

M. Steyvers and H. Tejeda. Bayesian modeling of human-ai comple- mentarity, Oct 2023. URL osf.io/2ntrf

work page 2023
[23]

Tariq, M

S. Tariq, M. B. Chhetri, S. Nepal, and C. Paris. A2c: A modular multi- stage collaborative decision framework for human-ai teams.arXiv preprint arXiv:2401.14432, 2024

work page arXiv 2024
[24]

A. A. Tutul, T. Chaspari, S. I. Levitan, and J. Hirschberg. Human-ai collaboration for the detection of deceptive speech. In11th Interna- tional Conference on Affective Computing and Intelligent Interaction, ACII 2023 - Workshops and Demos, Cambridge, MA, USA, September 10-13, 2023, pages 1–4. IEEE, 2023. doi: 10.1109/ACIIW59127.2023. 10388114. URL https:...

work page doi:10.1109/aciiw59127.2023 2023
[25]

Venanzi, J

M. Venanzi, J. Guiver, G. Kazai, P. Kohli, and M. Shokouhi. Community-based bayesian aggregation models for crowdsourcing. In Proceedings of the 23rd international conference on World wide web, pages 155–164, 2014

work page 2014
[26]

Verma and E

R. Verma and E. Nalisnick. Calibrated learning to defer with one-vs-all classifiers. InInternational Conference on Machine Learning, pages 22184–22202. PMLR, 2022

work page 2022
[27]

J. Wu, Z. Huang, Z. Hu, and C. Lv. Toward human-in-the-loop ai: En- hancing deep reinforcement learning via real-time human guidance for autonomous driving.Engineering, 21:75–91, 2023

work page 2023
[28]

X. Wu, L. Xiao, Y . Sun, J. Zhang, T. Ma, and L. He. A survey of human-in-the-loop for machine learning.Future Generation Computer Systems, 135:364–381, 2022

work page 2022
[29]

Zhang, K

Z. Zhang, K. Wells, and G. Carneiro. Learning to complement with mul- tiple humans (lecomh): Integrating multi-rater and noisy-label learning into human-ai collaboration.arXiv preprint arXiv:2311.13172, 2023

work page arXiv 2023