{"id":"56f01661-53e9-45f6-9b9e-5fa8bddb4ac9","arxiv_id":"2508.06997","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"A greedy, instance-specific expert-subset selection method based on conformal prediction sets achieves near-optimal classification accuracy with multiple human experts on CIFAR-10H and ImageNet-16H.","lead":"This paper proposes a greedy algorithm that picks, for each image, a small subset of human experts to label it, using prediction sets from a machine learning model. It reports near-optimal accuracy on real human labels from CIFAR-10H and ImageNet-16H, beating naive expert-selection baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim presumes conformal sets encode instance-specific expert reliability; this correlation is the unverified load-bearing step.","rationale":"The reader's weakest_assumption—that model conformal sets and human expert competence are related—is exactly the point on which the central claim stands or falls. The abstract provides no direct evidence of that correlation, and the full text provided is not legible enough to verify whether the paper includes a control. I agree with UNVERDICTED: the evidence is insufficient to accept, conditionally accept, or reject. The permutation test above is the minimal experiment that would turn UNVERDICTED into CONDITIONAL (if the gain is sensitive to the permutation) or REJECT (if the gain is invariant), because it directly tests whether the conformal set carries instance-specific information rather than only marginal coverage.","tokens_in":19889,"tokens_out":5742,"duration_ms":61153,"concrete_test":"Permute the conformal sets across test instances while preserving their sizes (and hence marginal coverage), then rerun the greedy subset-selection pipeline and the final accuracy evaluation on CIFAR-10H/ImageNet-16H. If the performance gap over naive baselines does not shrink materially, the conformal sets carry no instance-specific routing signal and the central claim is unsupported. As a secondary check, compute exhaustive optimal subsets on a small held-out slice using final classification accuracy as the objective and compare with the greedy objective used in the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's mechanism is that a prediction falling inside the conformal set C(x) makes that expert worth selecting for x. Conformal calibration alone only ensures the true label is in C(x) with probability at least 1-alpha; it imposes no relationship between C(x) and which humans are competent on x. The greedy algorithm's advantage therefore depends entirely on an empirical dependence between model uncertainty and expert reliability. The abstract's 'only certain experts may be relevant for each instance' asserts this dependence, but the reported simulations comparing greedy selection with naive baselines do not isolate the effect. If the apparent gain survives when the instance-specific link between C(x) and expert competence is broken, the improved performance could come from set size or marginal coverage rather than from routing; if the gain disappears, the central claim is supported. Without this control, 'near-optimal subsets' is also relative to an objective that may not equal final classification accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a conformal-set-based greedy algorithm for selecting, at the instance level, a subset of human experts whose predictions are used for classification. The central claim is that only certain experts are relevant for each instance, and that conformal prediction sets from a pre-trained model can guide this selection. The abstract reports that the method outperforms naive subset-selection baselines and achieves near-optimal subsets, with simulations based on real expert predictions from CIFAR-10H and ImageNet-16H. The supplied full text, however, is severely corrupted and mostly unreadable, so the formal problem definition, the greedy algorithm, the theoretical characterization, and the detailed experimental protocol cannot be verified from the manuscript as provided.","tokens_in":20111,"tokens_out":3548,"duration_ms":40750,"significance":"If the result holds, the paper would make a useful practical contribution to human-AI complementarity: it addresses a natural multi-expert extension of decision-support systems and evaluates on real human-expert datasets, which is a genuine strength. The basic idea of using conformal sets for expert routing is plausible and worth investigating. However, the key assumption that conformal uncertainty sets encode instance-specific expert competence is not established by conformal calibration alone, and the abstract provides no evidence for it. The significance of the claimed near-optimality also depends on the definition and practicality of the optimal-subset baseline. These issues are load-bearing, so the contribution is currently plausible but unverified.","major_comments":[{"comment":"The central claim presumes that membership of an expert's prediction in the conformal set C(x) identifies experts that are competent on x. Conformal prediction only provides marginal coverage of the true label; it does not by itself imply that C(x) is informative about human expert reliability. The paper needs (a) a direct quantitative measurement of the relationship between C(x) membership and per-expert correctness, and (b) a control experiment in which C(x) is replaced by a matched-size set with the same marginal coverage but no instance-specific signal. Without such a control, the reported improvement over naive methods could be driven by set size or dataset-specific base rates rather than by effective routing.","section":"Abstract, load-bearing mechanism"},{"comment":"The abstract does not specify the objective that the greedy algorithm is claimed to be near-optimal for, nor how the optimal subset is computed. If near-optimality is measured against a conformal-set objective, it does not necessarily imply improved final classification accuracy, especially since the aggregation rule for the selected experts' predictions is not stated. The experimental section, as far as legible, also does not provide error bars or a clear comparison protocol. The authors should define the oracle subset, report accuracy against that oracle, and state the aggregation rule (e.g., majority vote) explicitly.","section":"Abstract, 'near-optimal subsets' comparison"},{"comment":"The supplied full text is severely corrupted: large portions are repeated gibberish, the equations are unreadable, and the text even contains a header from a different arXiv paper. Consequently, the greedy algorithm's formal definition, the 'characterization of conditions' promised in the abstract, and the simulation protocol cannot be checked. This is not a cosmetic issue because the paper's central technical contributions live in those sections. The manuscript must be resupplied in a clean, readable form before the claims can be accepted.","section":"Full Text, readability and verifiability"}],"minor_comments":[{"comment":"The phrase 'naive methods for human subset selection' is undefined. The paper should name the baselines (e.g., fixed subset, random subset, all experts, single-best expert) and describe how they are selected.","section":"Abstract, baselines"},{"comment":"Even in the legible portions, notation such as C(x), alpha, the number of experts, and the aggregation of selected predictions is not defined in a self-contained way. A notation table and a precise problem statement would help.","section":"Notation and definitions"},{"comment":"All table captions and numerical entries are garbled in the supplied text. The authors should check that the PDF/source compiles correctly and that every table and figure has an intact caption.","section":"Tables and figures"},{"comment":"The extraction includes a header for arXiv:2508.06995v2 [cs.CV], suggesting that an incorrect or corrupt source file was uploaded. Please verify that the correct manuscript file is submitted.","section":"Source file integrity"}],"recommendation":"major_revision","confidential_remarks":"The supplied manuscript text is too corrupt for a fair technical review. Before sending to referees again, the editorial office should request a clean, compiled PDF. The scientific question is interesting and the use of real expert data is a plus, but the missing control experiment on the conformal-set/expert-competence link is essential; without it, the central mechanism is not established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Fair warning before anything else: the full text you sent me is a garbled PDF extraction — lines like '����������' — so I can't read the derivations, algorithms, or results tables. My assessment rests on the abstract and a few fragmentary sentences. So this is a provisional read, not a verdict.\n\nWhat looks genuinely new: the paper defines instance-specific subset selection from a pool of experts, instead of the usual single-expert decision-support setting, and uses conformal sets to drive the routing. That's a legitimate and practical extension. Two concrete strengths: they test on real expert predictions (CIFAR-10H, ImageNet-16H), not simulated labelers, and they compare against naive subset methods. If the greedy selection holds up, routing a fraction of instances to a small subset of experts while keeping accuracy is a useful result.\n\nThe soft spot is the one the stress-test note flags, and it's in the center of the claim. Conformal calibration guarantees marginal coverage, not that C(x) identifies which experts are reliable for x. The paper's insight — 'only certain experts may be relevant for each instance' — is an empirical assertion, and the abstract doesn't show the correlation between model uncertainty and expert competence. Nor does it describe the control that would break the correlation: if the gain persists when the link is destroyed, the improved performance could come from set size or coverage, not from intelligent routing. The term 'near-optimal' also needs a precise oracle; without that, we can't tell what the optimum is relative to. These are checks for the referee, not demonstrated flaws — but they are exactly where the paper lives or dies.\n\nThe corollary: the copy I have cannot be peer reviewed as-is. The encoding needs to be fixed, or the authors should provide a version with readable math and tables. The abstract alone doesn't give enough to verify the empirical protocol, error bars, or comparison baselines.\n\nBottom line: worth sending to a serious referee. The problem is well-formed, the data are real, and the central assumption is testable. I would ask the referee specifically to demand the routing-vs-set-size control and a definition of the 'optimal' oracle. I'm not citing it yet, and I wouldn't bring it to a reading group until the full text is readable. But it deserves a fair review.","headline":"Promising multi-expert routing idea built on conformal sets, but the unreadable full text and an untested correlation between conformal sets and expert competence keep it from being evaluable from this copy.","tokens_in":20490,"tokens_out":2662,"would_cite":false,"duration_ms":27674,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a greedy algorithm using conformal prediction sets can select, for each instance, the subset of human experts whose predictions should decide the label, and that this improves accuracy over naive expert selection.","keywords":["conformal prediction","human-AI complementarity","expert subset selection","greedy algorithm","decision support systems","multiple experts","CIFAR-10H","ImageNet-16H"],"falsifier":"Rerun the simulations with expert labels randomly permuted across instances while holding the conformal sets fixed. If the greedy rule's accuracy gain over a random subset does not vanish, the reported improvement is not caused by the set–expert correspondence; if it does vanish, that correspondence is the active mechanism.","tokens_in":19841,"feed_emoji":"🎯","tokens_out":6494,"duration_ms":59363,"temperature":0.7,"pith_summary":"This paper is about a decision-support setting with one pre-trained model and many human experts, and it asks a question the single-expert literature skips: which experts should be consulted for a given instance? The authors' central proposal is that the model's conformal prediction set — the small set of labels it regards as plausible — can act as a routing signal, because different human experts are reliable on different instances. They characterize conditions under which several experts can benefit from conformal sets, then introduce a greedy algorithm that builds an instance-specific expert subset by matching expert predictions to the set. Simulations with real expert predictions on CIFAR-10H and ImageNet-16H show the greedy subsets are near-optimal and beat naive subset-selection rules. If this holds, per-instance expert routing can be obtained almost for free from a model that already outputs conformal sets.","feed_headline":"Conformal sets pick the right human experts per image","feed_subtitle":"A greedy rule over model uncertainty sets nears optimal selection accuracy with real human labels.","key_machinery":"The central object is the conformal prediction set $\\mathcal{C}(x)$ produced by a pre-trained model: a calibrated set of labels that contains the true label with a chosen probability. The mechanism is a greedy subset-selection algorithm that treats these sets as a shared representation of what is uncertain about instance $x$; for each candidate expert subset, it scores how well the experts' predictions align with $\\mathcal{C}(x)$, adds the expert that improves the fit the most, and terminates according to a criterion tied to conformal coverage. The conformal set supplies the selection signal, and the greedy step keeps the search over $2^M$ expert subsets computationally manageable while pres","core_discovery":"The discovery claimed is that conformal prediction sets are not only uncertainty communication devices but also useful selection devices in a multi-expert human-AI system. Starting from the premise that only a subset of experts is relevant to each instance, the paper defines an optimization problem over expert subsets and solves it with a greedy algorithm: use the conformal set to score each expert's relevance, add the most relevant expert at each step, and stop when the selected panel covers the set well. The paper reports that this greedy rule achieves near-optimal subsets and better classification accuracy than naive baselines, in simulations built on real expert predictions from CIFAR-10","pith_inferences":["An extension the authors leave implicit is that the greedy rule is model-agnostic: any classifier with calibrated conformal sets could host it, making expert routing a plug-in layer rather than a separately trained component.","A testable prediction beyond the paper's simulations is that the selection gain should increase with expert specialization: pools with sharply different error profiles should show larger gains than homogeneous pools, since the conformal set separates them more cleanly.","The paper's simulations treat expert cost as uniform; if the conformal-set signal is as informative as claimed, a natural next step is to make subset size depend on a cost budget, asking when a second expert is worth consulting.","One could test the mechanism's limits by removing the conformal calibration: if the same greedy rule run on raw model confidence sets loses its edge, the property doing the work is the coverage guarantee rather than the model's score ordering."],"forward_implications":["A system can route each example to a small instance-specific panel of experts without training a separate router; the conformal set already carries the routing information.","Accuracy should beat both single-expert delegation and naive all-experts aggregation, because unhelpful experts are excluded on a per-example basis.","Because the same sets are already used to communicate uncertainty, adding expert selection to an existing conformal decision-support system costs little extra machinery.","Near-optimal subset performance in simulation indicates that selective consultation can cut expert workload while maintaining accuracy when the expert pool is large.","The stated conditions for multi-expert benefit provide a concrete diagnostic for whether expanding the expert pool will help a given conformal decision-support setup."],"supporting_citations":[],"fun_headline_variants":["Conformal sets pick the right expert per image","Greedy conformal selection finds near-optimal experts","Conformal sets decide which experts to consult","Selecting expert panels with conformal prediction","Conformal sets: the smart way to choose experts"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that a pre-trained model's conformal prediction sets are correlated with which human experts are reliable on each instance; if conformal uncertainty and expert competence are independent, the greedy rule has no signal to exploit and cannot beat random or fixed subset selection.","fun_headline_variants_meta":{"raw":{"variants":["Conformal sets pick the right expert per image","Greedy conformal selection finds near-optimal experts","Conformal sets decide which experts to consult","Selecting expert panels with conformal prediction","Conformal sets: the smart way to choose experts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00032,"raw_usage":{"total_tokens":1610,"prompt_tokens":685,"completion_tokens":925,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":429,"completion_tokens_details":{"reasoning_tokens":853}},"tokens_in":429,"tokens_out":925,"duration_ms":9050,"temperature":1.0,"reasoning_tokens":853,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:23:14.850010+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the simulations with expert labels randomly permuted across instances while holding the conformal sets fixed. If the greedy rule's accuracy gain over a random subset does not vanish, the reported improvement is not caused by the set–expert correspondence; if it does vanish, that correspondence is the active mechanism.","supporting_citations":[],"review_version":1}