{"id":"eb9bd442-6a45-48f4-98de-b8f96322e3ce","arxiv_id":"2608.11275","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"A pooling rule is only correct when the numbers it combines are generated in a matching way, and every rule's mismatch has a measurable probability and decision cost.","lead":"This paper works out which formula to use when two numbers both estimate the answer to the same yes/no question. Each common rule, from simple averaging to multiplying odds, is matched to a specific situation and the cost of using the wrong rule is measured.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified: the known-mechanism limitation is real but disclosed, and it does not undermine the paper's conditional exactness claims.","rationale":"The reader identified the known-mechanism requirement as the weakest assumption, and I agree that it is the most consequential limitation. However, the paper's core contribution is a set of conditional exactness statements with explicitly constructed generative worlds. Those statements are proved by elementary derivations and checked by reproducible simulations, and the practical limitation is openly disclosed rather than hidden. The known-mechanism requirement does not create an error in the conditional claims; it only separates the paper's conditional mapping from an unconditional prescription for real applications. Given the paper's clear scope and the reproducibility of its experiments, the ACCEPT verdict is appropriate and no adjustment is needed.","tokens_in":27414,"tokens_out":10491,"duration_ms":97401,"concrete_test":"Run the shipped deterministic suite exactly as documented, i.e. 'python3 run_all.py | diff - expected_output.txt', and confirm that all reported oracle gaps, frequencies, and Murphy-diagram curve values match the checked-in expected output. As an additional analytical check, independently re-derive the World 9 oracle probability from the rejection-sampling construction by computing P(G1=1, G2=0) / P(not both decisive) plus half the neither-decisive probability, and verify that this equals the DS pignistic BetP formula for the same m1, m2 values.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is explicitly conditional: a rule is exact in a world when it returns the conditional probability of the outcome from the information supplied to the rule, and each studied rule is paired with a constructed world in which this holds. The weakest point is the requirement that a practitioner know the generating mechanism and its parameters in advance. This is stated in Limitations: 'The experiments assume that the applicable generating mechanism and its parameters are known; the paper does not study how to infer them from data.' That limitation narrows the practical scope of the 'matching' contribution, but it does not invalidate any derivation or simulation. The exactness results are analytic: for example, Proposition 2 derives the MYCIN conflict value from Poisson thinning, the GK formula c1 + c2 - c1c2/s follows from inclusion-exclusion over identified shared activation events, and the Dempster-Shafer world is exactly rejection sampling conditioned on non-conflict. The simulations are deterministic and reproducible, and the mismatched-rule costs are measured relative to the true conditional probability, not to an arbitrarily chosen metric. I find no internal inconsistency or hidden assumption that breaks the conditional central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies the problem of combining two numerical assessments of the same yes/no question. It distinguishes several input types (posterior probabilities, event probabilities, confidence strengths, belief masses, and proofs with shared premises) and, for each of a dozen combination rules, constructs a generative world in which the rule equals the true conditional probability of the outcome given the supplied reports. The matching worlds include mixture-of-readings for averaging, conditionally independent evidence for odds pooling, doubled log-odds for geometric pooling, independent attempts for the probabilistic sum, Poisson thinning for the MYCIN rule, independent opposition for the ProbLog rule, rejection sampling on non-conflict for Dempster-Shafer, and identified shared premises for the GK calculation. Monte Carlo simulations confirm the analytic probabilities, and a second set of experiments measures the logarithmic-score and cost-loss consequences of using a mismatched rule. The paper also proves a decision-equivalence result at threshold 1/2 and shows that pairwise collapse of proof probabilities loses information for three or more derivations.","tokens_in":27622,"tokens_out":16013,"duration_ms":119633,"significance":"The paper's contribution is a clear, explicit mapping from data-generating assumptions to combination rules, with a formal notion of exactness (returning the conditional probability from the supplied information). The derivations are elementary but useful; Proposition 2 (Poisson thinning for the MYCIN conflict formula) is a neat and correct result. The experiments are reproducible: the public repository includes deterministic expected output, and the reported Monte Carlo frequencies agree with the analytic values within stated standard errors. The cost-loss and Murphy-diagram analysis connects proper scoring to decision costs carefully. If the conditional-exactness claims are taken as intended, with the known-mechanism assumption stated in the Limitations, the paper is a sound reference for practitioners choosing among pooling rules.","major_comments":[{"comment":"The displayed prior-corrected odds-pool formula, 'combined odds = o1 o2 / (1−π)/π', is algebraically incorrect as written. Since each reported posterior odds oi equals the likelihood ratio times the prior odds O0 = π/(1−π), the correct combined odds are o1·o2/O0 = o1·o2·(1−π)/π (equivalently o1·o2/(π/(1−π))). The text's formula would multiply by the prior odds rather than divide them out, contradicting the sentence 'the correct combination divides the doubled prior odds back out once' and the World 2′ results in Section 4.2. The code bayes_odds_pool in Appendix A.1 implements the correct version; please fix the equation and any derived statements.","section":"Section 2.5"}],"minor_comments":[{"comment":"Even after correcting the denominator, the notation 'o1o2 / (1−π)/π' is ambiguous; please write the corrected formula with explicit parentheses, e.g., o1·o2·(1−π)/π or o1·o2/(π/(1−π)).","section":"Section 2.5"},{"comment":"The cross-rule tables apply MYCIN and ProbLog to probability reports as if they were confidence values, which is not their intended input type; the paper acknowledges this in Section 2.12, but a local footnote or table caption would prevent misreading of the comparisons in worlds 1–5.","section":"Section 4.2"},{"comment":"In the 'Three proofs' paragraph, please specify the exact event sets for the three chains (for example, A={a1,a2}, B={a2,a3}, C={a3,a4} with additional independent items) so that the reader can verify the stated values 0.8984, 0.8318, and 0.9169 from the inclusion–exclusion calculation.","section":"Section 3.8"},{"comment":"The legend entries in the provided text are duplicated and cluttered ('oracle minimum, average, geometric, upco, mycin, noisy-or' appears in two parts of the caption); please clean up the legend and ensure it matches the plotted curves.","section":"Figure 5"},{"comment":"The 'How to read the tables' note says 'column order decreasing realized mean log growth', but the tables in the text do not clearly label each column; adding short column headers would improve readability.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is methodologically sound and the conditional claims are honestly scoped. The main issue is the inverted prior-odds formula in Section 2.5, which is clearly a typo given the code and simulations; I have no concerns about reproducibility or novelty. The paper fits the journal's scope well. I recommend major revision mainly to ensure the formula is corrected and the cross-rule tables are clarified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read this as a synthesis paper with a few genuinely new pieces, and it is better than the genre usually is. The central idea — that each pooling rule is exact in a specific constructed generative world — is worked out with elementary derivations and then checked with deterministic, seeded Monte Carlo code that ships with a commit hash. That is real evidence, and it matches the analytic claims. Proposition 1 (which rules tie at threshold 1/2) is simple but useful, and Proposition 2's Poisson-thinning derivation of the MYCIN conflict value is a nice touch that gives the rule a clean probabilistic reading. The GK treatment of overlapping proofs, with the explicit shared-premise calculation and the demonstration that pairwise folding loses information for three or more proofs, is the most novel part and is solid as far as I checked.\n\nThe soft spots are real but proportionate. The biggest is stated by the authors themselves in the Limitations section: the experiments assume the generating mechanism and its parameters are known, and the paper does not study how to infer them. That narrows the practical advice — a practitioner who cannot identify the world does not get an unconditional rule choice. I don't see this as a flaw in the conditional claims, which are the paper's actual contribution, but it does mean the title's 'matching rules to situations' is a map of worlds, not an estimation procedure. Second, much of the component material is not new: upco, geometric pooling, and noisy-or are all textbook, and the paper leans on that literature. The novelty is in the systematic pairing of worlds to rules and in the specific generative constructions, not in the formulas themselves. That is fine, but readers should not expect an open mathematical question to be resolved.\n\nThe evaluation methodology is a genuine strength. The logarithmic-score/betting framing is clearly explained, the oracle-gap metric is sensible, and the cost-loss analysis with Murphy diagrams is a good way to show that binary accuracy at one threshold hides real differences. The Schervish identity connecting the two is checked numerically to four decimals, which is the kind of reproducibility I like.\n\nWho gets value from this? Practitioners who combine confidence scores from LLM pipelines, retrieval systems, or proof procedures will find the world-rule correspondence clarifying, and researchers in forecast aggregation will find the generative perspective a useful complement to the axiomatic one. It deserves a serious referee, and with minor revision — mainly tightening the known-mechanism caveat in the abstract and introduction — it would be a solid publication. I would cite it.","headline":"A careful, reproducible map from generative worlds to pooling rules, with honest limits: the conditional claims hold up, and the paper deserves a serious referee despite the known-mechanism caveat.","tokens_in":28157,"tokens_out":745,"would_cite":true,"duration_ms":20618,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62C05","62F15","91B06","68T37"],"pacs":[],"model":"deepseek-v4-flash","headline":"Every common pooling rule is exact only in its own generative world; outside that world the rule loses betting wealth and decision value.","keywords":["probability pooling","opinion pooling","multiplicative odds pooling","geometric pooling","MYCIN certainty factors","Dempster-Shafer theory","probabilistic logic","Kelly betting"],"falsifier":"Simulate a Poisson support process with rate $\\lambda_a$ and independent refutation probability $\\lambda_b/\\lambda_a$; if the frequency of at least one surviving supporting item deviates from $\\frac{a-b}{1-b}$ beyond Monte Carlo error, then Proposition 2's central identity for the MYCIN rule is false.","tokens_in":27210,"feed_emoji":"🎲","tokens_out":9612,"duration_ms":98162,"temperature":0.7,"pith_summary":"The paper argues that there is no universally correct formula for combining two probability or confidence reports: each common rule is exact only under a specific generative story about what the numbers mean and how the sources relate, and outside that story the rule is miscalibrated and loses decision value. It constructs, for each of ten rules, a world—a complete probability model for reports and outcome—in which the rule returns the conditional probability of the outcome, and verifies the derivations by Monte Carlo simulation. It then measures the cost of using a mismatched rule through Kelly betting (logarithmic score) and threshold cost-loss decisions, finding oracle gaps up to 0.381 bits per round. The practical payoff is a checklist: identify the input type, the source relation, and the opposition structure, and the rule follows. The paper's stated limitation is that the recipe assumes the generative mechanism and its parameters are known.","feed_headline":"Each pooling rule is exact in one world—and costly outside it","feed_subtitle":"Match the formula to the story behind the numbers; mismatches lose up to 0.38 bits per round.","key_machinery":"The central device is the notion of a world: a complete probability model that jointly generates the two reports and the outcome, with a rule called exact when its output equals the true conditional probability of the outcome given the supplied inputs. The argument's workhorses are conditional-expectation averaging for mixture worlds; the odds identity $o_1o_2$ (with a prior correction) for independent evidence; the Poisson-thinning identity for MYCIN, $\\lambda(x)=-\\ln(1-x)$ and $(a-b)/(1-b)=1-e^{-(\\lambda_a-\\lambda_b)}$; inclusion-exclusion over shared premise identities for proofs, $c_1+c_2-c_1c_2/s$; and Schervish's representation connecting the logarithmic score to weighted cost-loss threshold losses, which lets every mismatch be scored in bits per round.","core_discovery":"Each pooling formula is not a rival estimator of one underlying quantity; it is the exact conditional probability for a specific way of generating the two reports and the outcome. Averaging is exact when an unobserved variable picks one of two alternative readings; multiplicative odds pooling (upco) is exact when two calibrated posterior reports arise from conditionally independent evidence and a shared prior; geometric pooling is exact for doubled log-odds or shared evidence; the probabilistic sum is exact for independent attempts and the maximum for nested attempts; the MYCIN conflict rule is exact for item-level refutation, via the identity $\\frac{a-b}{1-b}=1-e^{-(\\lambda_a-\\lambda_b)}$; the ProbLog rule is exact for independent opposition; Dempster–Shafer values are exact for opposing sources conditioned on not both being decisive; and the GK calculation $c_1+c_2-c_1c_2/s$ is exact for proofs with identified shared premises. The paper also proves that four canonical rules make the same binary decision at threshold $1/2$ exactly when $p_1+p_2>1$, so binary accuracy at that threshold cannot distinguish them, while proper scoring and threshold costs can.","pith_inferences":["A natural next step the paper does not take is world identification: fit the candidate generative mechanisms to calibration data and then apply the corresponding rule, turning the paper's conditional claims into an actionable model-selection procedure.","Because the betting oracle gap equals an integral of threshold cost-loss expenses, a practitioner whose operational costs concentrate near one cost ratio $c$ can optimize the Murphy diagram at that threshold rather than the full log score.","The GK result suggests that pipeline designers should log shared premise identities, such as retrieved passages or sub-results, rather than only the combined confidence; this is testable in self-consistency-style decoding, where sampled chains often reuse the same passages.","The decision equivalence at threshold $1/2$ implies that leaderboards reporting only binary accuracy at that threshold are structurally blind to pooling-rule differences; adding a proper score or a Murphy diagram would reveal them."],"forward_implications":["In any system where one of several alternative interpretations applies, averaging is the correct combined probability, and rules that multiply evidence will be miscalibrated.","When two reports come from conditionally independent evidence with a common prior, odds multiplication with the prior correction is exact, while plain averaging under-reinforces and loses about 0.089 bits per round in the paper's world 2.","For sources that uniformly distort their log-odds by a factor $k$, a symmetric exponent $\\alpha=1/k$ in the weighted log-odds family is exactly calibrated; geometric pooling is the $k=2$ case.","For conflicting evidence, the generative relation fixes the rule: independent supporting and opposing searches give $a(1-b)$, item-level refutation gives $(a-b)/(1-b)$, and conditioning on non-conflict gives Dempster-Shafer quantities; the naive difference is exact in neither tested setting.","When three or more proofs share premises, retaining premise identities and applying inclusion-exclusion is necessary; pairwise folding of the two-proof formula produces incorrect union probabilities (0.8318 versus 0.8984 in the paper's chain example)."],"supporting_citations":[{"why":"Defines the linear opinion pool that the paper's average world formalizes and extends.","marker":"[Stone, 1961]"},{"why":"Supplies the upco formula for updating on the credences of others under independent evidence.","marker":"[Easwaran et al., 2016]"},{"why":"Gives the probabilistic interpretation of MYCIN certainty factors that underlies the item-level refutation world.","marker":"[Heckerman, 1986]"},{"why":"Introduces ProbLog, from which the independent-opposition rule and its proof-union semantics come.","marker":"[De Raedt et al., 2007]"},{"why":"Provides inference machinery for ProbLog and weighted Boolean formulas on which the proof-union calculations rely.","marker":"[Fierens et al., 2015]"},{"why":"Defines Dempster's rule and its normalization, which the paper's non-conflict conditioning world realizes.","marker":"[Dempster, 1967]"},{"why":"Supplies the evidence-theoretic framework for masses, belief, plausibility, and pignistic probability.","marker":"[Shafer, 1976]"},{"why":"Establishes the log-wealth interpretation of the logarithmic score used to measure rule mismatch costs.","marker":"[Kelly, 1956]"},{"why":"Provides the representation theorem connecting proper scoring rules to weighted threshold loss functions.","marker":"[Schervish, 1989]"},{"why":"Describes the GK calculation with shared uncertain premises that the paper evaluates as its world-10 exact rule.","marker":"[Tammet, 2026]"}],"fun_headline_variants":["Every pooling rule has its own exact world","Match the pooling rule to the situation or pay","Pooling formulas: each exact in its proper context","Your pooling rule should match your data story"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The mapping from situation to rule only applies when the practitioner already knows which generative mechanism produced the reports and knows its parameters; the paper explicitly does not study inferring the mechanism from data.","fun_headline_variants_meta":{"raw":{"variants":["Every pooling rule has its own exact world","Match the pooling rule to the situation or pay","Pooling formulas: each exact in its proper context","Your pooling rule should match your data story"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000474,"raw_usage":{"total_tokens":2385,"prompt_tokens":1005,"completion_tokens":1380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":1322}},"tokens_in":621,"tokens_out":1380,"duration_ms":9185,"temperature":1.0,"reasoning_tokens":1322,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:13:03.105685+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate a Poisson support process with rate $\\lambda_a$ and independent refutation probability $\\lambda_b/\\lambda_a$; if the frequency of at least one surviving supporting item deviates from $\\frac{a-b}{1-b}$ beyond Monte Carlo error, then Proposition 2's central identity for the MYCIN rule is false.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the linear opinion pool that the paper's average world formalizes and extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the upco formula for updating on the credences of others under independent evidence."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the probabilistic interpretation of MYCIN certainty factors that underlies the item-level refutation world."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces ProbLog, from which the independent-opposition rule and its proof-union semantics come."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides inference machinery for ProbLog and weighted Boolean formulas on which the proof-union calculations rely."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Dempster's rule and its normalization, which the paper's non-conflict conditioning world realizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the evidence-theoretic framework for masses, belief, plausibility, and pignistic probability."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the log-wealth interpretation of the logarithmic score used to measure rule mismatch costs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the representation theorem connecting proper scoring rules to weighted threshold loss functions."},{"cited_title":"Structure-Preserving Uncertainty Propagation in First-Order Proof Search","cited_arxiv_id":"2608.09190","evidence_quote":"Describes the GK calculation with shared uncertain premises that the paper evaluates as its world-10 exact rule."}],"review_version":1}