{"id":"1b262066-7527-423b-a85d-62aecf1008d0","arxiv_id":"2607.20529","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Multi-LLM ensembles can be made robust to weak or adversarial members by weighting each model by its per-subject log score on a small set of calibration questions, instead of trusting all models equally.","lead":"The paper weights each LLM in an ensemble by how well its probability estimates scored on a small set of questions with known answers, instead of trusting all models equally. This suppresses confidently-wrong and adversarial members, keeping accuracy high on heterogeneous or contaminated panels while reducing overconfident errors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Verbalised-probability elicitation is never validated as a trustworthy score source; if per-model distortion differs, Cooke weights are artefacts and the claimed accuracy-reliability advantage is unsupported.","rationale":"The strongest claim is that Cooke weighting achieves a superior accuracy-reliability balance and is robust to unreliable experts. The entire mechanism is the softmax of per-subject log scores (Eq. 5). If those log scores are computed from verbalised probabilities that are not faithful, the weights are not estimates of trust. The paper cites evidence of miscalibrated verbalised confidence (Xiong et al. 2023) and acknowledges the risk in Limitations, but does not test it. This is more load-bearing than the missing error bars in Tables 3-4: even with error bars, the comparison would only be valid if the score source measures what is claimed. The proposed test—replacing the probability source or calibrating it, and checking whether the Cooke advantage survives—would settle whether the result is an artifact. I agree with the reader's weakest assumption; the seed-target distribution issue is secondary given the paper's two-stage design with 20/80 splits. The reader's CONDITIONAL verdict is appropriate; if the concrete test is run and fails, the verdict should move toward REJECT.","tokens_in":14783,"tokens_out":5627,"duration_ms":64857,"concrete_test":"On the existing 20/80 splits, recompute Cooke weights using (a) token log-probabilities over the answer choices instead of verbalised probabilities, and (b) verbalised probabilities after per-model temperature scaling fitted on the seed set; then re-evaluate Tables 2-3. If the Cooke advantage over global/accuracy weighting shrinks or reverses under either variant, the method is an artifact of the elicitation prompt. Also compute per-expert Spearman correlation between seed log scores and target-set NLL/accuracy within each subject; a weak or negative correlation means Eq. 5 is not measuring reliability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. 4 computes seed log scores from the verbalised distributions p_i(y|x), and Eq. 5 turns those scores into trust weights. The paper never shows that these verbalised numbers are comparable across models or that they track true reliability. Xiong et al. (2023), cited by the authors, shows verbalised confidence is often miscalibrated, and the Limitations admit 'inaccuracies in probability estimates may affect weighting.' If some models hedge toward uniform distributions while others overconfidently spike, the log score orders models by elicitation style, not by calibration, so Eq. 6 is not trust-weighted. Moreover, NLL/Brier/OE on target sets are computed from the same verbalised distributions, so the probabilistic metrics can reward whichever model produces the most 'convenient' numbers rather than the most reliable forecasts. The claimed superiority over global and accuracy weighting (Tables 2-3) therefore rests on an unvalidated construct. This is the load-bearing premise: if it fails, the central accuracy-reliability claim loses its foundation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper adapts Cooke's classical model from structured expert judgment to aggregate probabilistic predictions from multiple LLMs. It estimates subject-specific trust weights from held-out seed questions using log scores of verbalised probability distributions, then linearly pools experts with softmax-normalized weights (Eqs. 4-6). Experiments on MMLU and MMLU-Pro compare Cooke weighting against majority vote, equal averaging, global weighting, and accuracy weighting under homogeneous, heterogeneous, and contaminated panels, including subject-specific synthetic experts. The paper claims that Cooke weighting provides the best accuracy-reliability balance and remains robust when unreliable experts are introduced.","tokens_in":14853,"tokens_out":7532,"duration_ms":76822,"significance":"If the central premise is valid, the paper makes a useful contribution: it reframes LLM ensembling as trust estimation rather than prediction combination, and it provides a transparent held-out seed/target protocol with controlled synthetic contamination. Strengths include the clean separation of seed and target questions (§4.1), full tables with standard deviations (Tables 5-6), and ablations on seed size and temperature. However, the load-bearing verbalised-probability construct is not validated, and the headline comparisons lack statistical testing. The result is plausible but not yet fully established.","major_comments":[{"comment":"The method's core input is the verbalised probability distribution p_i(y|x) elicited by the prompt in §A.2. The paper treats these as faithful expert forecasts, but it also cites Xiong et al. (2023) showing verbalised confidence is often miscalibrated, and the Limitations acknowledge that 'inaccuracies in probability estimates may affect weighting.' No evidence is given that per-model elicitation distortions are small or comparable. If some models hedge toward uniform distributions while others overconfidently spike, the log scores in Eq. (4) rank by elicitation style, Eq. (5) turns this into weights, and Eq. (6) is not trust-weighted. Because NLL/Brier/OE are computed from the same verbalised distributions, the reported probabilistic advantages could be artifacts. Please validate the construct: per-model reliability diagrams, comparison with token-level probabilities on a subset, or agr","section":"§3.2, §A.2, Eqs. (4)-(6)"},{"comment":"The headline 'substantially outperforming' claim is not backed by significance tests or confidence intervals. For example, Table 2 shows Cooke MMLU accuracy 94.27 vs global weighting 93.66 and equal averaging 92.10; with only 5 splits and std devs of 0.17/0.28, the first gap may be within noise. OE differences such as 1.65 vs 1.68 vs 1.82 are very small. Please report paired tests or bootstrap CIs across the 5 random splits for the key comparisons, or state which differences are not significant. Without this, 'superior accuracy-reliability balance' is not statistically substantiated.","section":"§4.1, Tables 2, 5, 6"},{"comment":"The robustness advantage is asserted from visual curves: no numeric accuracy values at contamination ratios, error bars, or confidence intervals are given in the main text. The synthetic experts are described only narratively ('random', 'overconfident-wrong', 'biased') without an exact algorithm for generating their probability distributions, so the contaminated-panel results are hard to reproduce or verify. Please provide a numeric table with means and CIs at each contamination ratio and specify the generative procedure for each noisy-expert type, including the specialist and corrupted experts used in Tables 3-4.","section":"§5.3, Figs. 2, 6"},{"comment":"The Limitations correctly note that if the seed set does not reflect the target distribution, weights may not generalise. Because the experiments use random within-subject splits, seed-target shift is absent by construction, so the context-aware advantage (Tables 3-4) may overstate deployment performance. This does not invalidate the controlled comparison, but §6's generalisation language should be softened, or a covariate-shift experiment (e.g., training seeds on a subset of subjects and testing on others) should be added.","section":"Limitations"}],"minor_comments":[{"comment":"The summation index is inconsistent: it writes log p_i(y_j|x_j) but D^c_seed is defined over j; please clarify indices.","section":"Eq. (4)"},{"comment":"The disagreement-based method of Kruse et al. (2025) is discussed and argued to be fragile, but no experiment includes this or any other modern uncertainty-aware ensemble baseline. Adding such a comparison would strengthen the paper's positioning.","section":"§2, §6"},{"comment":"Model version details and inference settings (e.g., decoding temperature, number of JSON parsing retries) are not specified, which limits reproducibility.","section":"Appendix A.2"},{"comment":"No code or data availability statement is provided. Releasing the evaluation scripts and the verbalised probability outputs would help verify the results.","section":"General"},{"comment":"The legend entry 'accuracy' appears to label the accuracy-weighting baseline; consider renaming it to 'accuracy weighting' for consistency.","section":"Figure 1"},{"comment":"Minor typo: 'In practise' should be 'In practice'.","section":"§3.2"},{"comment":"The abstract uses 'adversarial experts' but the synthetic experts are not adversarial in a game-theoretic sense; consider 'noisy or unreliable experts'.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the validity of verbalised probabilities; I believe this is fixable with a focused validation study rather than new large-scale benchmarks. The held-out design and controlled contamination are strengths, and the central claim is plausible. I would also encourage the editor to ask for significance testing and code/data release."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a straightforward adaptation of Cooke's expert weighting to multi-LLM aggregation, with per-subject weights as the genuinely new twist. The headline claim — that seed-based log-score weighting beats equal-trust aggregation under heterogeneity and contamination — is plausible and mostly supported by the experiments. The thing to worry about is the construct validity of the probabilities used to compute the weights.\n\nThe experimental design is a real strength. The held-out seed/target split is clean, the reporting is unusually honest (the appendix gives standard deviations, and the paper says outright that Cooke weighting is not uniformly dominant), and the robustness curves make intuitive sense. The subject-specific contamination experiments — specialists and corrupted experts — are the most interesting part: they directly test the context-aware mechanism, and Cooke weighting wins there. That is genuine evidence that per-subject weighting captures something global weighting misses.\n\nThe main soft spot is the elicitation premise. The paper instructs each LLM to 'Return calibrated probabilities' and treats those outputs as proper forecasts, yet it cites evidence (Xiong et al. 2023) that verbalised confidence is frequently miscalibrated, and the Limitations admit that inaccuracies in probability estimates may affect weighting. There is no validation that the elicited distributions are comparable across models or that they track true reliability. If some models hedge toward uniform and others overconfidently spike, the log scores in Eq. 4 order models by elicitation style, not calibration. That doesn't automatically sink the paper — the accuracy gains under heterogeneity are large enough that they'd likely survive some noise — but it does mean the probabilistic metrics (NLL, Brier, OE) could partly reward convenient distributions rather than genuine uncertainty quality. A validation experiment comparing verbalised probabilities against sampling-based or token-level likelihood estimates would go a long way.\n\nOther soft spots are minor: Tables 3-4 show no error bars while the appendix reports sigma of 0.2-0.4 in comparable settings, so some context-aware gains may be within noise; the synthetic specialists/corrupted experts are not specified enough to reproduce; no code or data is released; and the identical OE values across several methods in Tables 1 and 5 look odd and need clarification.\n\nWho should read this: people working on LLM ensembling and uncertainty calibration. It won't change aggregation theory — the math is Cooke's 1991 method — but it gives a simple, checkable baseline and a useful empirical comparison. I'd send it to peer review. The elicitation concern is real but addressable, and the experimental design is transparent enough that a referee can evaluate it.","headline":"A clean, honest application of Cooke's classical expert weighting to LLM ensembles; the per-subject twist is useful, but the paper never validates that the verbalised probabilities it trusts are trustworthy enough to carry the load.","tokens_in":15519,"tokens_out":2462,"would_cite":true,"duration_ms":28136,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM ensembles should weight each model by its seed-question log score, not trust it equally.","keywords":["LLM ensembles","trust estimation","log-score weighting","calibration questions","overconfident errors","uncertainty calibration","context-aware aggregation","MMLU-Pro"],"falsifier":"Take a heterogeneous panel and compute two sets of seed weights for the same experts: one from each model's verbalised probabilities, one from its token-level likelihoods (or temperature-scaled logits). If the two weight vectors rank the models differently on seed questions, or if the verbalised weights do not predict held-out seed performance, the calibration signal is an elicitation artefact and the central claim fails. A complementary check: add a strategically 'good on seeds, bad on targets' expert; if log-score weighting still outperforms equal averaging in that panel, the robustness clai","tokens_in":14467,"feed_emoji":"⚖️","tokens_out":8266,"duration_ms":81946,"temperature":0.7,"pith_summary":"This paper argues that combining several language models by equal trust is the wrong default: an ensemble should first estimate which models deserve trust, using a small set of calibration questions with known answers. It adapts a classical expert-judgement rule in which each model's weight is the exponential of its average log-probability on correct seed answers, computed per subject, and the final output is the weighted mixture of all models' probability distributions. On multiple-choice benchmarks the rule behaves like simpler aggregators when all models are strong, but once weak or deliberately noisy models join, it keeps accuracy high while producing far fewer high-confidence errors than majority voting or equal averaging. The paper treats this as evidence that uncertainty-aware trust estimation, not accuracy or global reliability alone, is what makes an LLM ensemble robust.","feed_headline":"Calibration scores make LLM ensembles robust to bad experts","feed_subtitle":"Seed-question log scores identify trustworthy models and keep the blend accurate even when bad experts are added.","key_machinery":"The mechanism has three parts: a logarithmic scoring rule, a softmax weighting, and a per-subject context split. For each subject c, every expert model i receives s_{i,c}, the average log probability it placed on the correct answers in that subject's seed questions; the model's trust weight is w_{i,c} = exp(tau s_{i,c}) / sum_m exp(tau s_{m,c}); and the ensemble prediction is the weighted mixture p_agg(y|x) = sum_i w_{i,c} p_i(y|x). The log score does the main work because it punishes confident wrong answers much more heavily than uncertain ones. The context split is what lets the same expert be trusted in one subject and ignored in another, which is the capability that global weighting lack","core_discovery":"The paper's central claim is that per-subject log-score trust weighting—giving each model a softmax weight derived from its average log probability on calibration questions in that subject—achieves the best accuracy-reliability balance among aggregation methods. The evidence is empirical: in heterogeneous and contaminated panels on MMLU and MMLU-Pro, the weighting rule attains the highest or near-highest accuracy while maintaining low overconfident-error rates, and it shifts weight away from noisy experts and toward subject specialists only inside their specialty. The mechanism is the log score's asymmetry: a model that assigns high probability to a wrong seed answer is penalised far more th","pith_inferences":["The paper leaves open whether the same log-score weighting would transfer to open-ended generation, where probability distributions are not naturally defined; a testable extension is to elicit probabilities over sampled continuations and compare.","The method's success depends on verbalised probabilities being faithful; one can probe this by recomputing seed weights from token-level likelihoods or logits and checking whether the model ranking survives.","An adversarial expert could hide its failure mode from seed questions by behaving well on them; constructing seed sets that deliberately stress known weaknesses would test whether the robustness claim holds under strategic contamination.","The temperature parameter tau controls trust concentration; the paper shows moderate values help and very large values hurt, so an adaptive tau selected on held-out calibration data is a natural next step."],"forward_implications":["Equal-weight and majority-vote ensembles are fragile once model quality varies; a calibration-weighted mixture degrades more slowly as unreliable experts are added.","Small seed sets already provide useful trust signals, and larger seed sets improve the per-subject weights, so calibration questions are a practical tool rather than a large data requirement.","Aggregation can reduce overconfident errors without sacrificing accuracy, which matters for high-stakes deployment where confident wrong answers are costly.","Subject-specific weighting recovers domain expertise: a specialist model is upweighted only in its domain, and a corrupted model is downweighted only where it fails.","Proper scoring rules, not raw accuracy, are the right basis for deciding which LLM to trust, because they distinguish confidently wrong from merely uncertain predictions."],"fun_headline_variants":["Trust the well-calibrated: log scores reign in rogue LLMs","Weight LLMs by calibration to beat bad experts","Log-score weighting foils unreliable LLM experts","Subject-wise trust scores keep LLM ensembles safe","Calibration beats naive averaging in LLM teams"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole scheme rests on two linked premises: that the probabilities each model gives on seed questions are faithful enough to rank models by true reliability, and that seed questions come from the same distribution as the questions the ensemble will be asked; if either fails, the weights become noise and the mixture is not trust-weighted.","fun_headline_variants_meta":{"raw":{"variants":["Trust the well-calibrated: log scores reign in rogue LLMs","Weight LLMs by calibration to beat bad experts","Log-score weighting foils unreliable LLM experts","Subject-wise trust scores keep LLM ensembles safe","Calibration beats naive averaging in LLM teams"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000528,"raw_usage":{"total_tokens":2370,"prompt_tokens":719,"completion_tokens":1651,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":1589}},"tokens_in":463,"tokens_out":1651,"duration_ms":10219,"temperature":1.0,"reasoning_tokens":1589,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T07:40:46.239231+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a heterogeneous panel and compute two sets of seed weights for the same experts: one from each model's verbalised probabilities, one from its token-level likelihoods (or temperature-scaled logits). If the two weight vectors rank the models differently on seed questions, or if the verbalised weights do not predict held-out seed performance, the calibration signal is an elicitation artefact and the central claim fails. A complementary check: add a strategically 'good on seeds, bad on targets' expert; if log-score weighting still outperforms equal averaging in that panel, the robustness clai","supporting_citations":[],"review_version":1}