REVIEW 3 major objections 4 minor 1 cited by
Conformal Set-based Human-AI Complementarity with Multiple Experts
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper claims that a greedy algorithm using conformal prediction sets can select, for each instance, the subset of human experts whose predictions should decide the label, and that this improves accuracy over naive expert selection.
desk verdict Promising multi-expert routing idea built on conformal sets, but the unreadable full text and an untested correlation between conformal sets and expert competence keep it from being evaluable from this copy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the conformal prediction set $\mathcal{C}(x)$ produced by a pre-trained model: a calibrated set of labels that contains the true label with a chosen probability. The mechanism is a greedy subset-selection algorithm that treats these sets as a shared representation of what is uncertain about instance $x$; for each candidate expert subset, it scores how well the experts' predictions align with $\mathcal{C}(x)$, adds the expert that improves the fit the most, and terminates according to a criterion tied to conformal coverage. The conformal set supplies the selection signal, and the greedy step keeps the search over $2^M$ expert subsets computationally manageable while pres
What would settle it
Rerun the simulations with expert labels randomly permuted across instances while holding the conformal sets fixed. If the greedy rule's accuracy gain over a random subset does not vanish, the reported improvement is not caused by the set–expert correspondence; if it does vanish, that correspondence is the active mechanism.
Extended reading notes
Core claim
The discovery claimed is that conformal prediction sets are not only uncertainty communication devices but also useful selection devices in a multi-expert human-AI system. Starting from the premise that only a subset of experts is relevant to each instance, the paper defines an optimization problem over expert subsets and solves it with a greedy algorithm: use the conformal set to score each expert's relevance, add the most relevant expert at each step, and stop when the selected panel covers the set well. The paper reports that this greedy rule achieves near-optimal subsets and better classification accuracy than naive baselines, in simulations built on real expert predictions from CIFAR-10
Load-bearing premise
The load-bearing premise is that a pre-trained model's conformal prediction sets are correlated with which human experts are reliable on each instance; if conformal uncertainty and expert competence are independent, the greedy rule has no signal to exploit and cannot beat random or fixed subset selection.
Editorial extensions
If this is right
- A system can route each example to a small instance-specific panel of experts without training a separate router; the conformal set already carries the routing information.
- Accuracy should beat both single-expert delegation and naive all-experts aggregation, because unhelpful experts are excluded on a per-example basis.
- Because the same sets are already used to communicate uncertainty, adding expert selection to an existing conformal decision-support system costs little extra machinery.
- Near-optimal subset performance in simulation indicates that selective consultation can cut expert workload while maintaining accuracy when the expert pool is large.
- The stated conditions for multi-expert benefit provide a concrete diagnostic for whether expanding the expert pool will help a given conformal decision-support setup.
Reading between the lines
- An extension the authors leave implicit is that the greedy rule is model-agnostic: any classifier with calibrated conformal sets could host it, making expert routing a plug-in layer rather than a separately trained component.
- A testable prediction beyond the paper's simulations is that the selection gain should increase with expert specialization: pools with sharply different error profiles should show larger gains than homogeneous pools, since the conformal set separates them more cleanly.
- The paper's simulations treat expert cost as uniform; if the conformal-set signal is as informative as claimed, a natural next step is to make subset size depend on a cost budget, asking when a second expert is worth consulting.
- One could test the mechanism's limits by removing the conformal calibration: if the same greedy rule run on raw model confidence sets loses its edge, the property doing the work is the coverage guarantee rather than the model's score ordering.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a conformal-set-based greedy algorithm for selecting, at the instance level, a subset of human experts whose predictions are used for classification. The central claim is that only certain experts are relevant for each instance, and that conformal prediction sets from a pre-trained model can guide this selection. The abstract reports that the method outperforms naive subset-selection baselines and achieves near-optimal subsets, with simulations based on real expert predictions from CIFAR-10H and ImageNet-16H. The supplied full text, however, is severely corrupted and mostly unreadable, so the formal problem definition, the greedy algorithm, the theoretical characterization, and the detailed experimental protocol cannot be verified from the manuscript as provided.
Significance. If the result holds, the paper would make a useful practical contribution to human-AI complementarity: it addresses a natural multi-expert extension of decision-support systems and evaluates on real human-expert datasets, which is a genuine strength. The basic idea of using conformal sets for expert routing is plausible and worth investigating. However, the key assumption that conformal uncertainty sets encode instance-specific expert competence is not established by conformal calibration alone, and the abstract provides no evidence for it. The significance of the claimed near-optimality also depends on the definition and practicality of the optimal-subset baseline. These issues are load-bearing, so the contribution is currently plausible but unverified.
major comments (3)
- [Abstract, load-bearing mechanism] The central claim presumes that membership of an expert's prediction in the conformal set C(x) identifies experts that are competent on x. Conformal prediction only provides marginal coverage of the true label; it does not by itself imply that C(x) is informative about human expert reliability. The paper needs (a) a direct quantitative measurement of the relationship between C(x) membership and per-expert correctness, and (b) a control experiment in which C(x) is replaced by a matched-size set with the same marginal coverage but no instance-specific signal. Without such a control, the reported improvement over naive methods could be driven by set size or dataset-specific base rates rather than by effective routing.
- [Abstract, 'near-optimal subsets' comparison] The abstract does not specify the objective that the greedy algorithm is claimed to be near-optimal for, nor how the optimal subset is computed. If near-optimality is measured against a conformal-set objective, it does not necessarily imply improved final classification accuracy, especially since the aggregation rule for the selected experts' predictions is not stated. The experimental section, as far as legible, also does not provide error bars or a clear comparison protocol. The authors should define the oracle subset, report accuracy against that oracle, and state the aggregation rule (e.g., majority vote) explicitly.
- [Full Text, readability and verifiability] The supplied full text is severely corrupted: large portions are repeated gibberish, the equations are unreadable, and the text even contains a header from a different arXiv paper. Consequently, the greedy algorithm's formal definition, the 'characterization of conditions' promised in the abstract, and the simulation protocol cannot be checked. This is not a cosmetic issue because the paper's central technical contributions live in those sections. The manuscript must be resupplied in a clean, readable form before the claims can be accepted.
minor comments (4)
- [Abstract, baselines] The phrase 'naive methods for human subset selection' is undefined. The paper should name the baselines (e.g., fixed subset, random subset, all experts, single-best expert) and describe how they are selected.
- [Notation and definitions] Even in the legible portions, notation such as C(x), alpha, the number of experts, and the aggregation of selected predictions is not defined in a self-contained way. A notation table and a precise problem statement would help.
- [Tables and figures] All table captions and numerical entries are garbled in the supplied text. The authors should check that the PDF/source compiles correctly and that every table and figure has an intact caption.
- [Source file integrity] The extraction includes a header for arXiv:2508.06995v2 [cs.CV], suggesting that an incorrect or corrupt source file was uploaded. Please verify that the correct manuscript file is submitted.
Circularity Check
No significant circularity: conformal sets and expert predictions are external inputs, and the greedy algorithm's performance is tested on held-out real data.
full rationale
The paper's claimed derivation chain is: (i) a pre-trained model produces conformal prediction sets C(x) with marginal coverage guarantees; (ii) human experts supply predictions on the same instances; (iii) a greedy algorithm selects an instance-specific subset of experts using C(x); (iv) classification performance is evaluated on real expert datasets (CIFAR-10H, ImageNet-16H) against naive baselines and an optimal-subset oracle. None of these steps reduces to the others by construction. The conformal sets are calibrated to the true label rather than to expert competence; expert predictions are external data; the greedy selection rule is an exploratory ansatz whose value is assessed empirically, not derived from the conformal property. The 'near-optimal' claim is an empirical comparison: even if the optimal baseline were defined by the same surrogate objective, the final accuracy is measured against true labels, so the central performance claim is not forced by definition. The paper's main weakness is that the correlation between conformal sets and instance-specific expert reliability is asserted rather than independently established, but that is a validity/correctness concern, not circularity. No load-bearing self-citations, imported uniqueness theorems, or fitted-parameters-renamed-as-predictions were identifiable in the text.
Assumptions & free parameters
assumptions (2)
- domain assumption Conformal prediction sets from the pre-trained model are valid (cover the true label with a specified probability).
- domain assumption The pool of human experts can be queried per instance and their predictions can be combined.
Cite this review
Pith. "Pith review of Conformal Set-based Human-AI Complementarity with Multiple Experts." pith.science (2026). https://pith.science/paper/2WYX6JMU
@misc{pith2026250806997,
author = {Pith},
title = {Pith review of: Conformal Set-based Human-AI Complementarity with Multiple Experts},
year = {2026},
howpublished = {\url{https://pith.science/paper/2WYX6JMU}},
note = {Machine review of arXiv:2508.06997}
}
read the original abstract
Decision support systems are designed to assist human experts in classification tasks by providing conformal prediction sets derived from a pre-trained model. This human-AI collaboration has demonstrated enhanced classification performance compared to using either the model or the expert independently. In this study, we focus on the selection of instance-specific experts from a pool of multiple human experts, contrasting it with existing research that typically focuses on single-expert scenarios. We characterize the conditions under which multiple experts can benefit from the conformal sets. With the insight that only certain experts may be relevant for each instance, we explore the problem of subset selection and introduce a greedy algorithm that utilizes conformal sets to identify the subset of expert predictions that will be used in classifying an instance. This approach is shown to yield better performance compared to naive methods for human subset selection. Based on real expert predictions from the CIFAR-10H and ImageNet-16H datasets, our simulation study indicates that our proposed greedy algorithm achieves near-optimal subsets, resulting in improved classification performance among multiple experts.
Forward citations
Cited by 1 Pith paper
-
Tree-Based Formalization of Multi-Agent Complementarity in Human-AI Interactions
A tree-based formalization proves that reliance-based HAIs cannot be complementary under a pointwise-min benchmark, reduces regression complementarity to distance minimization, and shows classification is obstructed f...
Reference graph
Works this paper leans on
-
[1]
���������� ���� ��������� ������������� ������� ��� �������� ������� �������� ������� ����������� ������ �� �� �� ��� ���� ������� ���� �� �������� ���� ������ ����� ���� �� �� ����� ����� �� ���� ������ ��������� �� �� ������� ���� ����� ����� ������ �� �� ���� ��� ����� ��� ��������� ��������� ������������ ���������� ���� ���� ���� ���������� �� �������...
work page Pith review arXiv 2025
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.