{"id":"a09e2d96-c89d-47a7-9af4-6ff331778d5b","arxiv_id":"2412.14142","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In multi-distribution learning, the minimax-optimal predictor is calibrated only for the worst-case distribution, creating calibration disparities across other distributions.","lead":"This theory paper analyzes calibration of predictors trained across multiple distributions (MDL). It finds that the optimal worst-case predictor is well-calibrated only on the most pessimistic distribution, so calibration errors vary across groups and the same prediction can mean different things for different populations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 4.1 only upper-bounds calibration error; it never rules out perfect calibration, and a symmetric MDL instance yields a max-entropy predictor perfectly calibrated for every Q. Corollary 4.3 overclaims a fundamental trade-off.","rationale":"The reader's verdict is CONDITIONAL, and my analysis agrees that the paper's central claim needs revision, but the most load-bearing gap is not the minimax assumption (which the paper states explicitly) but the inference from Proposition 4.1 to Corollary 4.3. Proposition 4.1's inequality is an upper bound on calibration error. It does not lower-bound the calibration error, and it cannot rule out zero calibration error when the upper bound is zero or loose. The proof that h* 'cannot be perfectly canonically calibrated' derives only the tautological inequality E_Q[H_l(Q*(y|x))] ≤ E_Q*[H_l(Q*(y|x))], which holds because Q* maximizes entropy. The symmetric binary example shows a concrete MDL instance in which the max-entropy predictor is perfectly calibrated for both distributions; hence Corollary 4.3's universal 'different meaning' and 'cannot be calibrated' statements are false. This does not destroy the paper's contribution--the upper bound and the possibility of disparity remain--but it requires changing the headline claim from 'fundamental' to 'possible/generic' and adding the counterexample as a caveat. Since this is a major but fixable revision, the reader's CONDITIONAL verdict stands; hence UNCHANGED.","tokens_in":16781,"tokens_out":17706,"duration_ms":148602,"concrete_test":"Analytic check: instantiate the binary MDL example above (X={x1,x2}, uniform marginals, Q1=(0.1,0.9), Q2=(0.9,0.1) in P(y=1|x), log loss). Compute Q* = argmax_{Q∈conv(Q)} E_Q[H_Shannon(Q(y|x))] and the resulting h*(x)=Q*(y|x). Verify canonical calibration: for i=1,2, check Q_i(y|h*=ν)=ν for every ν in the support of h*. Conclude whether the counterexample violates Corollary 4.3. If it does, the paper's central claim must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the MDL-optimal max-entropy predictor h*=Q*(y|x) cannot be perfectly canonically calibrated for every Q (Corollary 4.3) is not established and is false in general. Proposition 4.1 proves only an upper bound: E_Q[d_l(Q(y|h*),h*)] ≤ E_Q*[H_l(Q*(y|x))] - E_Q[H_l(Q(y|x))]. The right-hand side can vanish for Q ≠ Q* whenever the two distributions share the same generalized entropy. In that case the bound is compatible with, and actually forces, zero calibration error. The proof's attempt to show impossibility derives an inequality that is always true because Q* maximizes entropy, so no contradiction follows. Concrete counterexample: X={x1,x2}, Y={0,1}, uniform marginals, Q1(y=1|x1)=0.1, Q1(y=1|x2)=0.9, Q2(y=1|x1)=0.9, Q2(y=1|x2)=0.1, log loss. Both Q1 and Q2 have identical generalized entropy H(0.1). The max-entropy distribution over conv(Q) is the constant mixture Q*(y|x)≡0.5, hence h*≡0.5. For i=1,2, Q_i(y=1|h*=0.5)=Q_i(y=1)=0.5, so h* is perfectly calibrated for both distributions. Calibration errors are uniform (zero) and the forecast has the same meaning. Thus Corollary 4.3's universal 'cannot be calibrated' and 'different meaning' statements are false; the correct statement is a possibility result, not a fundamental limitation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the calibration properties of predictors obtained from multi-distribution learning (MDL). It argues that, at the MDL-optimal saddle point, the optimal predictor is the conditional distribution Q*(y|x) of the distribution Q* that maximizes the generalized entropy of the loss, and that this predictor cannot be perfectly canonically calibrated for all distributions in the set. The authors derive an upper bound on the calibration error for any Q in terms of a generalized entropy gap (Proposition 4.1), claim a non-uniform calibration disparity (Corollary 4.2), assert a fundamental calibration-refinement trade-off even at optimality (Corollary 4.3), and discuss decision-theoretic consequences (Proposition 4.4) and implications for distributionally robust optimization and fairness (Section 6).","tokens_in":17065,"tokens_out":4262,"duration_ms":39097,"significance":"If the main claims were correct, the paper would provide a useful warning about a fundamental limitation of MDL. The calibration error upper bound in Proposition 4.1 is a valid and instructive observation, and the paper is right that MDL can lead to calibration disparities. However, the central impossibility claim---that perfect calibration for all Q is impossible and that a calibration-refinement trade-off is fundamental---is not established and is in fact false in general, as shown by a symmetric counterexample. The correct statement is a possibility result: calibration failures can occur, but they need not. The paper also contains a circular argument in Proposition 4.4 and an unsupported corollary for DRO. The contribution is therefore significantly weaker than claimed, but the upper-bound result and the articulation of conditions under which disparities arise may be salvageable after substantial revision.","major_comments":[{"comment":"The claim that h* = Q*(y|x) 'cannot be perfectly canonically calibrated' for Q != Q* is false. Proposition 4.1 only gives an upper bound: E_Q[d_l(Q(y|h*), h*)] <= E_Q*[H_l(Q*(y|x))] - E_Q[H_l(Q(y|x))]. The right-hand side can be zero for Q != Q* whenever the two distributions have the same generalized entropy, and the bound then forces zero calibration error. A concrete counterexample is X={x1,x2}, Y={0,1}, uniform marginals, log loss, Q1(y=1|x1)=0.1, Q1(y=1|x2)=0.9, Q2(y=1|x1)=0.9, Q2(y=1|x2)=0.1. Both Q1 and Q2 have identical generalized entropy H(0.1), and the max-entropy distribution over their convex hull is Q*(y|x)=0.5 for both x. Thus h* is the constant 0.5, and for each Q_i the marginal label distribution is Q_i(y=1)=0.5, so Q_i(y|h*=0.5)=0.5 and h* is perfectly calibrated for both distributions. This directly contradicts the universal 'cannot be calibrated' statement and the 'different meaning' conclusion of Corollary 4.3. The correct statement is a possibility result, not a fundamental limitation.","section":"Section 4, Proposition 4.1 and Corollary 4.3"},{"comment":"The proof of the claimed calibration-refinement trade-off is not valid. It asserts 'a < b' where a = E_Q[H_l(Q(y|x))] and b = E_Q[l(y,h*)], but this inequality is false for Q = Q*, where h* = Q*(y|x) and a = b. The subsequent argument that a more informative predictor 'will be compensated by increase in the calibration error' is informal: the calibration-refinement decomposition alone does not imply that decreasing refinement forces an increase in calibration error, since the total risk need not be fixed. A formal derivation of a trade-off would require showing that the gap b - a is strictly positive for Q != Q* and that changes in refinement are absorbed by calibration error in a controlled way; the current proof does neither.","section":"Section 4, Corollary 4.3 proof"},{"comment":"Proposition 4.4 is circular as stated. The proof in B.2 shows that the decision rule delta is optimal on average for Q*, and then says 'If we further assume that Q* also maximizes the generalized entropy with respect to the cost function c, under this assumption it can be said that delta achieves the value sup_Q min_delta E_Q[c(delta(h(x)),y)]'. But the assumption that Q* maximizes generalized entropy for c is essentially the desired conclusion that the worst-case decision optimality holds over Q. Without independent conditions relating c to l and to the ambiguity set Q, the proposition merely restates the assumption that the cost function is 'consistent' with the loss. The manuscript needs to state sufficient conditions under which the saddle-point for l is also a saddle-point for c.","section":"Section 4, Proposition 4.4 and Appendix B.2"},{"comment":"Corollary 6.1 is not established. It claims that an ambiguity set defined by a divergence d that is k-Lipschitz with respect to the generalized entropy function 'guarantees robust and equitable decisions', but the proof only notes that |H_l(P)-H_l(Q)| <= k*epsilon bounds the entropy gap in the calibration error bound. This bounds the calibration error, not the decision-theoretic optimality claimed in Proposition 4.4. Moreover, the statement that 'every proper scoring loss function is associated with a Bregman divergence that satisfies this continuity property' is unsubstantiated: the Bregman divergence is on prediction space, not on the space of distributions, and Lipschitz continuity of the entropy with respect to that divergence would need proof. The remark that KL is Lipschitz with respect to Shannon entropy is not true on the full simplex, since the gradient of entropy is unbounded near the boundary.","section":"Section 6, Corollary 6.1"}],"minor_comments":[{"comment":"In the proof of Proposition 4.1, the displayed relation 'EQ[H_l(Q*(y|x))] <= EQ*[H_l(Q*(y|x))]' appears to contain a typo: the left argument should presumably be Q(y|h*(x)) or Q(y|x), not Q*(y|x). This makes the subsequent sentence difficult to interpret.","section":"Section 4, Proposition 4.1 proof"},{"comment":"Corollary 4.2 states that calibration errors form 'a non-increasing sequence converging to 0' with lim_{Q->Q*}, but no sequence is defined and no topology on the set of distributions Q is specified. As written, this is not a precise mathematical statement; it should be rephrased in terms of a chosen metric or family of distributions approaching Q*.","section":"Section 4, Corollary 4.2"},{"comment":"There are several presentation issues: the phrase 'there exists Q* that attains the said quantity sup inf = inf' in Proposition 3.2 is imprecise about the order of quantifiers; Figure 1's caption does not explain the plotted quantities; and the text switches between 'h*' and 'h' inconsistently in places. These should be corrected.","section":"General"},{"comment":"The related work in Section 5 mentions the concurrent work of Fröhlich and Williamson (2024c) but does not carefully delineate which of the paper's claims are novel relative to that work, especially concerning maximum generalized entropy predictors and limitations for arbitrary cost functions. A more detailed comparison would help the reader assess the novelty.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The central impossibility result is disproven by a simple two-distribution example, which is likely to be a serious problem for the paper's main thesis. The upper-bound result and the possibility of calibration disparity are valid but constitute a much weaker contribution than claimed. The authors should be asked to correct or substantially weaken Corollary 4.3, repair Proposition 4.4, and either prove or remove Corollary 6.1. The paper also does not fully address the concurrent work it cites. I would encourage a revision because the upper-bound framework is useful, but the current version overclaims in ways that cannot be ignored."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper studies calibration at the MDL optimum, where the predictor is the max generalized entropy distribution Q*. That setup is sensible, and the authors prove a correct bound: for any Q in the ambiguity set, the calibration error is at most the entropy gap between Q* and Q. That is worth having, and the paper deserves credit for asking the question and for citing the concurrent Fröhlich–Williamson work honestly.\n\nBut the paper overclaims. Proposition 4.1 is an upper bound, and it never rules out perfect calibration. The 'cannot be perfectly calibrated' sentence is false. Take X={x1,x2}, Y={0,1}, uniform marginals, and Q1,Q2 with swapped conditionals 0.1/0.9. Both have the same generalized entropy for log loss. The max-entropy predictor over their convex hull is the constant 0.5, and every mixture is perfectly calibrated: for any Q_alpha, P(y=1|h*=0.5)=0.5. So calibration errors are uniform and the forecast has the same meaning for all distributions. That directly contradicts the abstract's 'non-uniform calibration errors' and Corollary 4.3's 'different meaning.' The correct claim is a possibility: calibration disparity can occur, and the bound controls it, but it is not a fundamental limitation. The trade-off statement should be reworked as a Pareto-style statement: perfect calibration for all Q may force low refinement (as the constant predictor shows), not that miscalibration is unavoidable.\n\nOther soft spots: Corollary 4.2 asserts convergence to zero without a topology or proof. Corollary 4.3's proof is informal and doesn't establish the trade-off. Proposition 4.4's 'consistent' cost function is undefined, and as stated the worst-case optimality essentially assumes Q* also maximizes generalized entropy for the cost, which is close to the conclusion. Section 6's claim that KL is Lipschitz w.r.t. Shannon entropy is false; distributions with same entropy can have large KL. That needs a fix before publication.\n\nIf these are corrected to possibility results and the misleading claims are softened, the paper could be a useful caution for DRO and fairness practice. As is, the main qualitative headline is too strong. It deserves a serious referee, but a conditional one.","headline":"The calibration bound is real, but the 'fundamental' framing is not supported—a symmetric counterexample kills the universal impossibility.","tokens_in":17633,"tokens_out":5527,"would_cite":false,"duration_ms":47581,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The minimax-optimal predictor in multi-distribution learning is calibrated only for the maximum-entropy distribution; for others, calibration error is generic and a calibration-refinement trade-off persists.","keywords":["multi-distribution learning","calibration","generalized entropy","proper scoring losses","calibration-refinement trade-off","distributionally robust optimization","min-max fairness","decision-making"],"falsifier":"Compute per-distribution expected calibration error for the max-entropy MDL predictor on a simple two-distribution envelope (e.g., two Gaussian classes with different priors): the claim predicts errors that are nonzero for any $Q \\neq Q^*$ and scale with the generalized-entropy gap, so finding an envelope where all $Q$ have zero calibration error would falsify the central assertion.","tokens_in":16465,"feed_emoji":"⚖️","tokens_out":11241,"duration_ms":87567,"temperature":0.7,"pith_summary":"Multi-distribution learning (MDL) promises a single predictor that performs uniformly well over a set of distributions, with applications in robustness and fairness. This paper asks whether that predictor can also be trusted as a calibrated confidence measure for every distribution in the set. It shows that the minimax-optimal MDL predictor—the conditional distribution of the maximum-generalized-entropy distribution $Q^*$—is perfectly calibrated for $Q^*$ but generically cannot be for any other distribution $Q$, with calibration error bounded by the gap in generalized entropies. Consequently, calibration errors are non-uniform across distributions and there is an inherent calibration-refinement trade-off at optimality: improving refinement for one distribution is compensated by worse calibration. If correct, this means MDL's worst-case loss guarantee does not automatically translate into reliable, equitable decisions for arbitrary downstream costs.","feed_headline":"The best multi-distribution predictor can't be calibrated for all","feed_subtitle":"Even at optimal performance, calibration errors vary across groups; a forecast means different things to different groups","key_machinery":"The load-bearing object is the generalized entropy $H_\\ell(Q(y|x)) = \\inf_{h \\in \\Delta^{|Y|}} \\mathbb{E}_Q[\\ell(y,h(x))]$ of a proper scoring loss $\\ell$, together with the saddle point $(Q^*, h^*)$ of the MDL zero-sum game in which $Q^* = \\arg\\max_{Q\\in\\mathcal{Q}} H_\\ell(Q(y|x))$ and $h^*(x) = Q^*(y|x)$. The argument runs through the calibration-refinement decomposition $\\mathbb{E}_Q[\\ell(y,h(x))] = \\mathbb{E}_Q[d_\\ell(Q(y|h(x)), h(x))] + \\mathbb{E}_Q[H_\\ell(Q(y|h(x)))]$, where $d_\\ell$ is the Bregman divergence associated with $\\ell$; this identity converts the saddle-point property into bounds on per-distribution calibration error and exposes the trade-off.","core_discovery":"The paper's central claim is that the MDL minimax problem has a unique saddle point $(Q^*, h^*)$ where $Q^*$ maximizes the generalized entropy $H_\\ell(Q(y|x))$ over the distribution set and $h^*(x) = Q^*(y|x)$. At this optimal point, the predictor is perfectly canonically calibrated for $Q^*$ but, absent special distributional coincidences, not for any other $Q$ in the set. Using the calibration-refinement decomposition of proper scoring losses—risk equals expected Bregman divergence between $Q(y|h^*(x))$ and $h^*(x)$ plus expected generalized entropy—the paper bounds the calibration error for each $Q$ by $\\mathbb{E}_{Q^*}[H_\\ell(Q^*(y|x))] - \\mathbb{E}_Q[H_\\ell(Q(y|x))]$. This bound is zero only when $Q$ collapses to $Q^*$, so calibration errors form a non-uniform pattern across the envelope. The authors conclude that the same forecast has different conditional-label meaning under different distributions, and a decision rule built on the calibrated forecast is worst-case optimal only for cost functions consistent with the training loss.","pith_inferences":["Beyond the paper's claims: the trade-off suggests that post-processing an MDL predictor to be calibrated on one distribution will push miscalibration onto other distributions, so fixes like temperature scaling cannot remove the disparity; this is testable by measuring per-distribution calibration error before and after such post-processing.","A further consequence is that the choice of training loss in MDL is itself a fairness decision, because the set of downstream cost functions for which the worst-case guarantee survives is determined by which loss was used to identify the max-entropy distribution.","The bounds may transfer to risk-measure formulations such as CVaR, where the envelope is defined by density constraints; in that case one would predict that calibration disparity grows with the allowed density ratio, a statement that empirical studies on robust optimization could check.","One could view the entropy-gap bound as an uncertainty principle for MDL: the product of worst-case calibration error and worst-case refinement loss across the envelope has a positive lower bound whenever the envelope contains distributions with different generalized entropies."],"forward_implications":["For every distribution $Q$ in the envelope, the MDL-optimal predictor's calibration error is at most $\\mathbb{E}_{Q^*}[H_\\ell(Q^*(y|x))] - \\mathbb{E}_Q[H_\\ell(Q(y|x))]$, so the error vanishes only as $Q$ approaches $Q^*$.","A decision rule that thresholds the calibrated forecast is worst-case optimal over the whole set only when the downstream cost function is consistent with the training loss; otherwise some distributions will receive systematically suboptimal decisions.","In distributionally robust optimization, selecting an ambiguity-set divergence that is $\\kappa$-Lipschitz with respect to the generalized entropy bounds the calibration disparity across the set by $\\kappa\\epsilon$.","In min-max fairness over $k$ distinct populations, the optimal predictor can fall inside the convex hull of the population distributions, so it is calibrated for none of the $k$ groups and needs post-processing.","At the MDL optimum, any gain in refinement for one distribution is compensated by an increase in its calibration error, so no predictor can be simultaneously perfectly calibrated and maximally refined for every distribution in the set."],"supporting_citations":[{"why":"gives the decomposition of a proper scoring loss into calibration error plus refinement, which is the linchpin of the calibration bounds.","marker":"Bröcker, 2009"},{"why":"establishes that maximum generalized entropy is robust Bayes and supplies minimax conditions that justify identifying $Q^*$ as the MDL optimum.","marker":"Grünwald & Dawid, 2004"},{"why":"formulates MDL as a zero-sum game and the on-demand sampling framework that the paper's optimal-predictor analysis extends.","marker":"Haghtalab et al., 2022"},{"why":"characterizes proper scoring losses through generalized entropy functions, providing the Bregman divergence used to define calibration error.","marker":"Ovcharov, 2018"},{"why":"supplies the decision-theoretic result that a calibrated predictor is optimal for arbitrary costs in a single distribution, which Proposition 4.4 extends to the worst-case MDL setting.","marker":"Noarov & Roth, 2024"},{"why":"defines the generalized Bayes rule and score, the benchmark the paper uses to derive the attainable lower bound.","marker":"Fröhlich & Williamson, 2024a"},{"why":"frames distributionally robust optimization as the ambiguity-set MDL setting used in the consequences section.","marker":"Rahimian & Mehrotra, 2022"},{"why":"motivates min-max fairness across subpopulations, the fairness application the paper discusses.","marker":"Rothblum & Yona, 2021"}],"fun_headline_variants":["Optimal MDL predictor still mis-calibrates some groups","Bayes-optimal MDL leaves calibration errors uneven","Minimax multi-distribution model: perfect for one, off for others","Multi-distribution learning: calibration varies at optimum","Even optimal MDL has non-uniform calibration errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the minimax theorem applying to the MDL game—the distribution set compact and convex, the predictor free to output any label distribution, and the loss proper, convex, and continuous—so that the optimum is exactly the maximum-entropy predictor, and on the absence of distributional coincidences that would make other distributions calibrated with that predictor.","fun_headline_variants_meta":{"raw":{"variants":["Optimal MDL predictor still mis-calibrates some groups","Bayes-optimal MDL leaves calibration errors uneven","Minimax multi-distribution model: perfect for one, off for others","Multi-distribution learning: calibration varies at optimum","Even optimal MDL has non-uniform calibration errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1304,"prompt_tokens":919,"completion_tokens":385,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":304}},"tokens_in":535,"tokens_out":385,"duration_ms":4349,"temperature":1.0,"reasoning_tokens":304,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:26:28.605817+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute per-distribution expected calibration error for the max-entropy MDL predictor on a simple two-distribution envelope (e.g., two Gaussian classes with different priors): the claim predicts errors that are nonzero for any $Q \\neq Q^*$ and scale with the generalized-entropy gap, so finding an envelope where all $Q$ have zero calibration error would falsify the central assertion.","supporting_citations":[{"cited_title":"On-demand sampling: Learning optimally from multiple distributions","cited_arxiv_id":null,"evidence_quote":"formulates MDL as a zero-sum game and the on-demand sampling framework that the paper's optimal-predictor analysis extends."},{"cited_title":"Proper scoring rules and bregman divergence","cited_arxiv_id":null,"evidence_quote":"characterizes proper scoring losses through generalized entropy functions, providing the Bregman divergence used to define calibration error."},{"cited_title":"Calibration for decision making: A principled approach to trustworthy ml, 2024","cited_arxiv_id":null,"evidence_quote":"supplies the decision-theoretic result that a calibrated predictor is optimal for arbitrary costs in a single distribution, which Proposition 4.4 extends to the worst-case MDL setting."},{"cited_title":"Frameworks and results in distributionally robust optimization","cited_arxiv_id":null,"evidence_quote":"frames distributionally robust optimization as the ambiguity-set MDL setting used in the consequences section."}],"review_version":1}