Pith. sign in

REVIEW 1 major objections 5 minor 76 references

Every pooling rule has its world: matching probability combination rules to situations and stakes

T0 review · 1 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Every common pooling rule is exact only in its own generative world; outside that world the rule loses betting wealth and decision value.

desk verdict A careful, reproducible map from generative worlds to pooling rules, with honest limits: the conditional claims hold up, and the paper deserves a serious referee despite the known-mechanism caveat. read the letter →

arxiv 2608.11275 v1 pith:2YTQ66H4 submitted 2026-08-11 stat.ME cs.AI

Tanel Tammet , Priit Järv , Dirk Draheim This is my paper · ORCID
classification stat.MEcs.AI MSC 62C0562F1591B0668T37
keywords probabilitypoolingopinionmultiplicativeoddsgeometricMYCINcertaintyfactorsDempster-ShafertheoryprobabilisticlogicKellybetting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that there is no universally correct formula for combining two probability or confidence reports: each common rule is exact only under a specific generative story about what the numbers mean and how the sources relate, and outside that story the rule is miscalibrated and loses decision value. It constructs, for each of ten rules, a world—a complete probability model for reports and outcome—in which the rule returns the conditional probability of the outcome, and verifies the derivations by Monte Carlo simulation. It then measures the cost of using a mismatched rule through Kelly betting (logarithmic score) and threshold cost-loss decisions, finding oracle gaps up to 0.381 bits per round. The practical payoff is a checklist: identify the input type, the source relation, and the opposition structure, and the rule follows. The paper's stated limitation is that the recipe assumes the generative mechanism and its parameters are known.

What carries the argument

The central device is the notion of a world: a complete probability model that jointly generates the two reports and the outcome, with a rule called exact when its output equals the true conditional probability of the outcome given the supplied inputs. The argument's workhorses are conditional-expectation averaging for mixture worlds; the odds identity $o_1o_2$ (with a prior correction) for independent evidence; the Poisson-thinning identity for MYCIN, $\lambda(x)=-\ln(1-x)$ and $(a-b)/(1-b)=1-e^{-(\lambda_a-\lambda_b)}$; inclusion-exclusion over shared premise identities for proofs, $c_1+c_2-c_1c_2/s$; and Schervish's representation connecting the logarithmic score to weighted cost-loss threshold losses, which lets every mismatch be scored in bits per round.

What would settle it

Simulate a Poisson support process with rate $\lambda_a$ and independent refutation probability $\lambda_b/\lambda_a$; if the frequency of at least one surviving supporting item deviates from $\frac{a-b}{1-b}$ beyond Monte Carlo error, then Proposition 2's central identity for the MYCIN rule is false.

Watch

Extended reading notes

Core claim

Each pooling formula is not a rival estimator of one underlying quantity; it is the exact conditional probability for a specific way of generating the two reports and the outcome. Averaging is exact when an unobserved variable picks one of two alternative readings; multiplicative odds pooling (upco) is exact when two calibrated posterior reports arise from conditionally independent evidence and a shared prior; geometric pooling is exact for doubled log-odds or shared evidence; the probabilistic sum is exact for independent attempts and the maximum for nested attempts; the MYCIN conflict rule is exact for item-level refutation, via the identity $\frac{a-b}{1-b}=1-e^{-(\lambda_a-\lambda_b)}$; the ProbLog rule is exact for independent opposition; Dempster–Shafer values are exact for opposing sources conditioned on not both being decisive; and the GK calculation $c_1+c_2-c_1c_2/s$ is exact for proofs with identified shared premises. The paper also proves that four canonical rules make the same binary decision at threshold $1/2$ exactly when $p_1+p_2>1$, so binary accuracy at that threshold cannot distinguish them, while proper scoring and threshold costs can.

Load-bearing premise

The mapping from situation to rule only applies when the practitioner already knows which generative mechanism produced the reports and knows its parameters; the paper explicitly does not study inferring the mechanism from data.

Editorial extensions

If this is right

  • In any system where one of several alternative interpretations applies, averaging is the correct combined probability, and rules that multiply evidence will be miscalibrated.
  • When two reports come from conditionally independent evidence with a common prior, odds multiplication with the prior correction is exact, while plain averaging under-reinforces and loses about 0.089 bits per round in the paper's world 2.
  • For sources that uniformly distort their log-odds by a factor $k$, a symmetric exponent $\alpha=1/k$ in the weighted log-odds family is exactly calibrated; geometric pooling is the $k=2$ case.
  • For conflicting evidence, the generative relation fixes the rule: independent supporting and opposing searches give $a(1-b)$, item-level refutation gives $(a-b)/(1-b)$, and conditioning on non-conflict gives Dempster-Shafer quantities; the naive difference is exact in neither tested setting.
  • When three or more proofs share premises, retaining premise identities and applying inclusion-exclusion is necessary; pairwise folding of the two-proof formula produces incorrect union probabilities (0.8318 versus 0.8984 in the paper's chain example).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the paper does not take is world identification: fit the candidate generative mechanisms to calibration data and then apply the corresponding rule, turning the paper's conditional claims into an actionable model-selection procedure.
  • Because the betting oracle gap equals an integral of threshold cost-loss expenses, a practitioner whose operational costs concentrate near one cost ratio $c$ can optimize the Murphy diagram at that threshold rather than the full log score.
  • The GK result suggests that pipeline designers should log shared premise identities, such as retrieved passages or sub-results, rather than only the combined confidence; this is testable in self-consistency-style decoding, where sampled chains often reuse the same passages.
  • The decision equivalence at threshold $1/2$ implies that leaderboards reporting only binary accuracy at that threshold are structurally blind to pooling-rule differences; adding a proper score or a Murphy diagram would reveal them.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. This paper studies the problem of combining two numerical assessments of the same yes/no question. It distinguishes several input types (posterior probabilities, event probabilities, confidence strengths, belief masses, and proofs with shared premises) and, for each of a dozen combination rules, constructs a generative world in which the rule equals the true conditional probability of the outcome given the supplied reports. The matching worlds include mixture-of-readings for averaging, conditionally independent evidence for odds pooling, doubled log-odds for geometric pooling, independent attempts for the probabilistic sum, Poisson thinning for the MYCIN rule, independent opposition for the ProbLog rule, rejection sampling on non-conflict for Dempster-Shafer, and identified shared premises for the GK calculation. Monte Carlo simulations confirm the analytic probabilities, and a second set of experiments measures the logarithmic-score and cost-loss consequences of using a mismatched rule. The paper also proves a decision-equivalence result at threshold 1/2 and shows that pairwise collapse of proof probabilities loses information for three or more derivations.

Significance. The paper's contribution is a clear, explicit mapping from data-generating assumptions to combination rules, with a formal notion of exactness (returning the conditional probability from the supplied information). The derivations are elementary but useful; Proposition 2 (Poisson thinning for the MYCIN conflict formula) is a neat and correct result. The experiments are reproducible: the public repository includes deterministic expected output, and the reported Monte Carlo frequencies agree with the analytic values within stated standard errors. The cost-loss and Murphy-diagram analysis connects proper scoring to decision costs carefully. If the conditional-exactness claims are taken as intended, with the known-mechanism assumption stated in the Limitations, the paper is a sound reference for practitioners choosing among pooling rules.

major comments (1)
  1. [Section 2.5] The displayed prior-corrected odds-pool formula, 'combined odds = o1 o2 / (1−π)/π', is algebraically incorrect as written. Since each reported posterior odds oi equals the likelihood ratio times the prior odds O0 = π/(1−π), the correct combined odds are o1·o2/O0 = o1·o2·(1−π)/π (equivalently o1·o2/(π/(1−π))). The text's formula would multiply by the prior odds rather than divide them out, contradicting the sentence 'the correct combination divides the doubled prior odds back out once' and the World 2′ results in Section 4.2. The code bayes_odds_pool in Appendix A.1 implements the correct version; please fix the equation and any derived statements.
minor comments (5)
  1. [Section 2.5] Even after correcting the denominator, the notation 'o1o2 / (1−π)/π' is ambiguous; please write the corrected formula with explicit parentheses, e.g., o1·o2·(1−π)/π or o1·o2/(π/(1−π)).
  2. [Section 4.2] The cross-rule tables apply MYCIN and ProbLog to probability reports as if they were confidence values, which is not their intended input type; the paper acknowledges this in Section 2.12, but a local footnote or table caption would prevent misreading of the comparisons in worlds 1–5.
  3. [Section 3.8] In the 'Three proofs' paragraph, please specify the exact event sets for the three chains (for example, A={a1,a2}, B={a2,a3}, C={a3,a4} with additional independent items) so that the reader can verify the stated values 0.8984, 0.8318, and 0.9169 from the inclusion–exclusion calculation.
  4. [Figure 5] The legend entries in the provided text are duplicated and cluttered ('oracle minimum, average, geometric, upco, mycin, noisy-or' appears in two parts of the caption); please clean up the legend and ensure it matches the plotted curves.
  5. [Section 4.2] The 'How to read the tables' note says 'column order decreasing realized mean log growth', but the tables in the text do not clearly label each column; adding short column headers would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all rule exactness claims are conditional and derived from explicitly constructed generative worlds, with no fitted parameters masquerading as predictions.

full rationale

The paper's central claim is explicitly conditional: a rule is exact in a world when it returns the conditional probability of the outcome from the information supplied to the rule. Each world is built from primitive events—mixture indicators for alternative readings, conditionally independent signals for upco, doubled log-odds for geometric pooling, Poisson arrivals with item-level refutation for MYCIN, independent activation events for GK, and decisive-evidence draws conditioned on non-conflict for Dempster–Shafer—and the target probability is computed from those primitives, not by inverting the rule. The Monte Carlo simulations fix seeds, generate outcomes from the primitives, and compare observed frequencies against analytically derived formulas; the paper explicitly states that 'the formulas are derived analytically; the simulations check their implementation and measure the errors of mismatched rules.' No parameter is fitted to make a rule match: the cumulation parameter a is an input swept over all values, the log-odds exponents follow from stated distortion factors, and the GK formula is evaluated on event sets whose probabilities are supplied. The only practical limitation—that the generating mechanism and its parameters must be known in advance—is disclosed in the Limitations section and narrows scope without making the conditional derivations circular. Self-citations to the authors' CONFER cumulation and GK work are not load-bearing: the count-based cumulation parameter is shown to be approximate (error up to 0.021 in Section 3.8), and the GK formula is independently simulated outside GK's proof search.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical or ontological entities. The 'worlds' are stochastic generative models defined in terms of standard probability objects (mixtures, independent signals, Poisson processes, rejection sampling). The free parameters listed are inputs to the rules being analyzed, supplied by the experiment design rather than fitted to make the central claims true; the paper states that mechanism identification and parameter estimation are outside its scope.

free parameters (4)
  • cumulation degree of independence a
    User-supplied overlap parameter of the cumulation rule (Section 2.9.1); the rule is exact under the all-or-nothing mixture model for every a, and approximate for item-level sharing (Section 3.7), so no single value is fitted.
  • case-mixture weights w_i
    Probabilities of the alternative readings in the weighted-average world (Sections 3.1, 4.5); treated as known inputs, not estimated.
  • prior odds correction pi
    Common prior required by the odds pool (Section 2.5); supplied as a known input in world 2'.
  • log-odds exponent alpha and asymmetric weights w1,w2
    Coefficients of the weighted log-odds family swept in Section 4.5; the exact corrections are derived from stated distortion factors, and the best value for partial sharing is found empirically but identified as approximate.
assumptions (6)
  • standard math Standard probability and inclusion-exclusion identities, Frechet bounds, and Poisson process thinning
    Used in Sections 2.4, 3.6, 3.8 and 4.4 as unproved background results.
  • domain assumption The two reports are generated by a known, fully specified stochastic mechanism ('world')
    Section 3 states every experiment is a controlled generating mechanism. The paper's recipe for choosing rules presumes the practitioner can identify this mechanism; the paper states in Limitations that it does not study how to infer it from data.
  • domain assumption Conditional independence of the two observations given the outcome, with a common prior, for the multiplicative/odds-pool world
    Section 3.4: each model receives a separate conditionally independent signal; the posterior combining formula is exact only under this dependence structure.
  • domain assumption Distinct GK activation events (uncertain ground premises) are independent
    Section 2.9.2: 'The model assumes that distinct activation events are independent.' Load-bearing for the GK union probability.
  • domain assumption Opposing evidence operates as item-level refutation (MYCIN world), as independent counter-evidence (ProbLog world), or under non-conflict conditioning (Dempster-Shafer world)
    Sections 3.6, 3.3 and 3.9 define the generative models that make each conflicting-evidence rule exact.
  • domain assumption Symmetric multiplicative distortion of log-odds or complete duplication of evidence for the geometric pool
    Sections 3.5 and 4.5: geometric pooling is exact when both sources double their log-odds or repeat the same evidence; these are the generative conditions stated as inputs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Every pooling rule has its world: matching probability combination rules to situations and stakes." pith.science (2026). https://pith.science/paper/2YTQ66H4

@misc{pith2026260811275,
  author       = {Pith},
  title        = {Pith review of: Every pooling rule has its world: matching probability combination rules to situations and stakes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2YTQ66H4}},
  note         = {Machine review of arXiv:2608.11275}
}
read the original abstract

Systems often need to combine two numerical assessments of the same yes/no question. The appropriate formula depends on what the numbers represent and on how the sources are related. Averaging is correct when one of several alternative interpretations applies; multiplying odds is correct when probability reports are based on conditionally independent evidence and a common prior; and probabilities of alternative successful derivations require their dependence or shared evidence to be taken into account. We state the assumptions behind several common combination rules and derive the corresponding combined probabilities. Two groups of Monte Carlo experiments address different questions. First, controlled generating mechanisms verify that the derived rule recovers the correct probability in the situations for which its assumptions hold. Second, the same mechanisms measure the consequences of using a mismatched rule, using logarithmic score and threshold decisions with different costs. Distinct pooling rules can produce the same binary decision at threshold 1/2 while assigning substantially different probabilities, so binary accuracy alone can conceal important differences. We also give probabilistic interpretations of conflicting-evidence rules and show that, for overlapping derivations, retaining the identities of shared uncertain premises permits direct calculation of the probability that at least one derivation is available. Pairwise combination of proof probabilities loses information when there are three or more derivations.

Figures

Figures reproduced from arXiv: 2608.11275 by the authors.

Figure 1
Figure 1. The probability-scale pools on the same inputs: pooled value against the first input [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. The confidence-scale combinators (0.5 = no information) with the second input fixed [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Observed frequency of the event against the pooled value (25 equal-count bins per rule) [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: One cumulative log-wealth path per pooling rule over 500 rounds at even odds (bits; 0 [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]
Figure 5
Figure 5. Figure 5: Murphy diagrams for the alternative-reading world (left; exact rule: the average) and [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]
Figure 6
Figure 6. Figure 6: The oracle gap (excess log loss) across the symmetric log-odds exponent [PITH_FULL_IMAGE:figures/full_fig_p026_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 71 canonical work pages

  1. [1]

    Acz\'el, J., Wagner, C. (1980). A characterization of weighted arithmetic means. SIAM Journal on Algebraic and Discrete Methods 1(3), 259--260

    work page 1980

  2. [2]

    Baccelli, J., Stewart, R. T. (2023). Support for geometric pooling. Review of Symbolic Logic 16(1), 298--337

    work page 2023

  3. [3]

    A., Tetlock, P

    Baron, J., Mellers, B. A., Tetlock, P. E., Stone, E., Ungar, L. H. (2014). Two reasons to make aggregated probability forecasts more extreme. Decision Analysis 11(2), 133--145

    work page 2014

  4. [4]

    Benavoli, A., Antonucci, A. (2010). An aggregation framework based on coherent lower previsions: application to Zadeh's paradox and sensor networks. International Journal of Approximate Reasoning 51(9), 1014--1028

    work page 2010

  5. [5]

    G., Shortliffe, E

    Buchanan, B. G., Shortliffe, E. H. (1984). Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project. Addison-Wesley

    work page 1984

  6. [6]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. O., Kaplan, J., et al. (2021). Evaluating large language models trained on code. arXiv:2107.03374

    arXiv 2021

  7. [7]

    Cooke, R. M. (1991). Experts in Uncertainty: Opinion and Subjective Probability in Science. Oxford University Press

    work page 1991

  8. [8]

    M., Thomas, J

    Cover, T. M., Thomas, J. A. (2006). Elements of Information Theory, 2nd ed. Wiley

    2006

Show all 76 references
  1. [9]

    Dempster, A. P. (1967). Upper and lower probabilities induced by a multivalued mapping. Annals of Mathematical Statistics 38(2), 325--339

    1967

  2. [10]

    Den ux, T. (2019). Decision-making with belief functions: a review. International Journal of Approximate Reasoning 109, 87--110

    2019

  3. [11]

    De Raedt, L., Kimmig, A., Toivonen, H. (2007). ProbLog: a probabilistic Prolog and its application in link discovery. In Proc.\ IJCAI 2007, 2468--2473

    2007

  4. [12]

    Dietrich, F. (2010). Bayesian group belief. Social Choice and Welfare 35(4), 595--626

    2010

  5. [13]

    Dietrich, F., List, C. (2016). Probabilistic opinion pooling. In: H\'ajek, A., Hitchcock, C. (eds.), The Oxford Handbook of Probability and Philosophy. Oxford University Press

    2016

  6. [14]

    Easwaran, K., Fenton-Glynn, L., Hitchcock, C., Velasco, J. D. (2016). Updating on the credences of others: disagreement, agreement, and synergy. Philosophers' Imprint 16(11), 1--39

    2016

  7. [15]

    Ehm, W., Gneiting, T., Jordan, A., Kr\"uger, F. (2016). Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings. Journal of the Royal Statistical Society B 78(3), 505--562

    2016

  8. [16]

    Fierens, D., Van den Broeck, G., Renkens, J., Shterionov, D., Gutmann, B., Thon, I., Janssens, G., De Raedt, L. (2015). Inference and learning in probabilistic logic programs using weighted Boolean formulas. Theory and Practice of Logic Programming 15(3), 358--401

    2015

  9. [17]

    Fissler, T., Lorentzen, C., Mayer, M. (2022). Model comparison and calibration assessment: user guide for consistent scoring functions in machine learning and actuarial practice. arXiv:2202.12780

    2022 arXiv

  10. [18]

    Geifman, Y., El-Yaniv, R. (2017). Selective classification for deep neural networks. In Advances in Neural Information Processing Systems 30, 4878--4887

    2017

  11. [19]

    Genest, C. (1984). A characterization theorem for externally Bayesian groups. Annals of Statistics 12(3), 1100--1105

    1984

  12. [20]

    Genest, C., Zidek, J. V. (1986). Combining probability distributions: a critique and an annotated bibliography. Statistical Science 1(1), 114--135

    1986

  13. [21]

    Gneiting, T., Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102(477), 359--378

    2007

  14. [22]

    Grant, A., Johnstone, D. (2010). Finding profitable forecast combinations using probability scoring rules. International Journal of Forecasting 26(3), 498--510

    2010

  15. [23]

    J., Karvounarakis, G., Tannen, V

    Green, T. J., Karvounarakis, G., Tannen, V. (2007). Provenance semirings. In Proc.\ PODS 2007, 31--40

    2007

  16. [24]

    Guo, C., Pleiss, G., Sun, Y., Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proc.\ ICML 2017, PMLR 70, 1321--1330

    2017

  17. [25]

    Hagedorn, R., Smith, L. A. (2009). Communicating the value of probabilistic forecasts with weather roulette. Meteorological Applications 16(2), 143--155

    2009

  18. [26]

    Hailperin, T. (1986). Boole's Logic and Probability, 2nd ed. North-Holland

    1986

  19. [27]

    Heckerman, D. (1986). Probabilistic interpretations for MYCIN's certainty factors. In: Kanal, L. N., Lemmer, J. F. (eds.), Uncertainty in Artificial Intelligence. North-Holland, 167--196

    1986

  20. [28]

    E., Shortliffe, E

    Heckerman, D. E., Shortliffe, E. H. (1992). From certainty factors to belief networks. Artificial Intelligence in Medicine 4(1), 35--52

    1992

  21. [29]

    Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation 14(8), 1771--1800

    2002

  22. [30]

    A., Madigan, D., Raftery, A

    Hoeting, J. A., Madigan, D., Raftery, A. E., Volinsky, C. T. (1999). Bayesian model averaging: a tutorial. Statistical Science 14(4), 382--417

    1999

  23. [31]

    Irving, G., Christiano, P., Amodei, D. (2018). AI safety via debate. arXiv:1805.00899

    2018 arXiv

  24. [32]

    A., Jordan, M

    Jacobs, R. A., Jordan, M. I., Nowlan, S. J., Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural Computation 3(1), 79--87

    1991

  25. [33]

    Johnstone, D. J. (2007). Economic Darwinism: who has the best probabilities? Theory and Decision 62(1), 47--96

    2007

  26. [34]

    J., Jose, V

    Johnstone, D. J., Jose, V. R. R., Winkler, R. L. (2012). Log-optimal economic evaluation of probability forecasts. Journal of the Royal Statistical Society A 175(3), 661--689

    2012

  27. [35]

    Jose, V. R. R., Grushka-Cockayne, Y., Lichtendahl, K. C. (2014). Trimmed opinion pools and the crowd's calibration problem. Management Science 60(2), 463--475

    2014

  28. [36]

    J., Uhlmann, J

    Julier, S. J., Uhlmann, J. K. (1997). A non-divergent estimation algorithm in the presence of unknown correlations. In Proc.\ American Control Conference 1997, 2369--2373

    1997

  29. [37]

    Kelly, J. L. (1956). A new interpretation of information rate. Bell System Technical Journal 35(4), 917--926

    1956

  30. [38]

    Kimmig, A., Van den Broeck, G., De Raedt, L. (2017). Algebraic model counting. Journal of Applied Logic 22, 46--62

    2017

  31. [39]

    Kingman, J. F. C. (1993). Poisson Processes. Oxford University Press

    1993

  32. [40]

    Lakshminarayanan, B., Pritzel, A., Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 30, 6402--6413

    2017

  33. [41]

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33, 9459--9474

    2020

  34. [42]

    Liao, F., Liang, M., Li, Z., Hu, X., Song, S. (2019). Evaluate the malignancy of pulmonary nodules using the 3D deep leaky noisy-or network. IEEE Transactions on Neural Networks and Learning Systems 30(11), 3484--3495

    2019

  35. [43]

    Lyu, Q., Shridhar, K., Malaviya, C., Zhang, L., Elazar, Y., Tandon, N., Apidianaki, M., Sachan, M., Callison-Burch, C. (2025). Calibrating large language models with sample consistency. In Proc.\ AAAI 2025, 39(18), 19260--19268

    2025

  36. [44]

    McConway, K. J. (1981). Marginalization and linear opinion pools. Journal of the American Statistical Association 76(374), 410--414

    1981

  37. [45]

    Murphy, A. H. (1977). The value of climatological, categorical and probabilistic forecasts in the cost-loss ratio situation. Monthly Weather Review 105(7), 803--816

    1977

  38. [46]

    Neiswanger, W., Wang, C., Xing, E. (2014). Asymptotically exact, embarrassingly parallel MCMC. In Proc.\ UAI 2014, 623--632

    2014

  39. [47]

    Nelsen, R. B. (2006). An Introduction to Copulas, 2nd ed. Springer

    2006

  40. [48]

    Nilsson, N. J. (1986). Probabilistic logic. Artificial Intelligence 28(1), 71--87

    1986

  41. [49]

    V., Lakshminarayanan, B., Snoek, J

    Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., Snoek, J. (2019). Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems 32, 1...

    2019

  42. [50]

    Pettigrew, R., Weisberg, J. (2023). Geometric pooling: a user's guide. The British Journal for the Philosophy of Science, doi:10.1086/727000

    2023 doi

  43. [51]

    Pettigrew, R., Weisberg, J. (2025). Jeffrey pooling. Philosophers' Imprint 25(8), 1--16, doi:10.3998/phimp.3806

    2025 doi

  44. [52]

    Radzvilas, M., Peden, W., De Pretis, F. (2023). Making decisions with evidential probability and objective Bayesian calibration inductive logics. International Journal of Approximate Reasoning 162, 109030

    2023

  45. [53]

    Ranjan, R., Gneiting, T. (2010). Combining probability forecasts. Journal of the Royal Statistical Society B 72(1), 71--91

    2010

  46. [54]

    Richardson, D. S. (2000). Skill and relative economic value of the ECMWF ensemble prediction system. Quarterly Journal of the Royal Meteorological Society 126(563), 649--668

    2000

  47. [55]

    Rivera, M., Godbout, J.-F., Rabbany, R., Pelrine, K. (2024). Combining confidence elicitation and sample-based methods for uncertainty quantification in misinformation mitigation. In Proc.\ UncertaiNLP 2024 (EACL Workshop on Uncertainty-Aware NLP), 114--126

    2024

  48. [56]

    S., Smith, L

    Roulston, M. S., Smith, L. A. (2002). Evaluating probabilistic forecasts using information theory. Monthly Weather Review 130(6), 1653--1660

    2002

  49. [57]

    A., Baron, J., Foster, D

    Satop\"a\"a, V. A., Baron, J., Foster, D. P., Mellers, B. A., Tetlock, P. E., Ungar, L. H. (2014). Combining multiple probability predictions using a simple logit model. International Journal of Forecasting 30(2), 344--356

    2014

  50. [58]

    Schervish, M. J. (1989). A general method for comparing probability assessors. Annals of Statistics 17(4), 1856--1879

    1989

  51. [59]

    L., Blocker, A

    Scott, S. L., Blocker, A. W., Bonassi, F. V., Chipman, H. A., George, E. I., McCulloch, R. E. (2016). Bayes and big data: the consensus Monte Carlo algorithm. International Journal of Management Science and Engineering Management 11(2), 78--88

    2016

  52. [60]

    Shafer, G. (1976). A Mathematical Theory of Evidence. Princeton University Press

    1976

  53. [61]

    H., Buchanan, B

    Shortliffe, E. H., Buchanan, B. G. (1975). A model of inexact reasoning in medicine. Mathematical Biosciences 23(3--4), 351--379

    1975

  54. [62]

    Smets, P., Kennes, R. (1994). The transferable belief model. Artificial Intelligence 66(2), 191--234

    1994

  55. [63]

    T., Ojea Quintana, I

    Stewart, R. T., Ojea Quintana, I. (2018). Probabilistic opinion pooling with imprecise probabilities. Journal of Philosophical Logic 47(1), 17--45

    2018

  56. [64]

    Stone, M. (1961). The opinion pool. Annals of Mathematical Statistics 32(4), 1339--1342

    1961

  57. [65]

    Tammet, T., Draheim, D., J\"arv, P. (2021). Confidences for commonsense reasoning. In Proc.\ CADE 28, LNCS 12699, Springer, 507--524

    2021

  58. [66]

    Tammet, T., Draheim, D., J\"arv, P. (2022). GK: implementing full first-order default logic for commonsense reasoning (system description). In Proc.\ IJCAR 2022, LNCS 13385, Springer

    2022

  59. [67]

    Tammet, T. (2026). Structure-preserving uncertainty propagation in first-order proof search. arXiv:2608.09190

    2026 arXiv

  60. [68]

    Thorne, J., Vlachos, A., Christodoulopoulos, C., Mittal, A. (2018). FEVER: a large-scale dataset for fact extraction and VERification. In Proc.\ NAACL-HLT 2018, 809--819

    2018

  61. [69]

    Tresp, V. (2000). A Bayesian committee machine. Neural Computation 12(11), 2719--2741

    2000

  62. [70]

    van Melle, W. (1980). A Domain-Independent System that Aids in Constructing Knowledge-Based Consultation Programs. PhD dissertation, Stanford University, Report STAN-CS-80-820

    1980

  63. [71]

    Verga, P., Hofst\"atter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., Lewis, P. (2024). Replacing judges with juries: evaluating LLM generations with a panel of diverse models. arXiv:2404.18796

    2024 arXiv

  64. [72]

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D. (2023). Self-consistency improves chain of thought reasoning in language models. In Proc.\ ICLR 2023

    2023

  65. [73]

    Wilks, D. S. (2001). A skill score based on economic value for probability forecasts. Meteorological Applications 8(2), 209--219

    2001

  66. [74]

    G., Izmailov, P

    Wilson, A. G., Izmailov, P. (2020). Bayesian deep learning and a probabilistic perspective of generalization. In Advances in Neural Information Processing Systems 33, 4697--4708

    2020

  67. [75]

    Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., Hooi, B. (2024). Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs. In Proc.\ ICLR 2024

    2024

  68. [76]

    Zadeh, L. A. (1984). Review of ``A Mathematical Theory of Evidence''. AI Magazine 5(3), 81--83

    1984

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.