Pith. sign in

REVIEW 4 major objections 5 minor 5 references

MIST: Mutual Information Estimation Via Supervised Training

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fully empirical, supervised meta-training approach can build practical mutual information estimators that outperform classical and neural baselines in low-sample, high-dimensional regimes—provided the training distribution covers the targ

desk verdict MIST is a solid, honestly-limited demonstration that supervised meta-training can beat KSG in low-sample synthetic regimes, but the OoMD generalization and calibration claims are both overstated relative to the paper's own appendix. read the letter →

arxiv 2511.18945 v4 pith:2YAJ7I5T submitted 2025-11-24 cs.LG cs.ITmath.IT

German Gritsai , Megan Richards , Maxime Méloux , Kyunghyun Cho , Maxime Peyrard This is my paper · ORCID
classification cs.LGcs.ITmath.IT
keywords mutualinformationestimationmeta-learningsupervisedlearningpermutation-invariantnetworksquantileregressionuncertaintyquantificationSetTransformersyntheticdistributions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that mutual information (MI) estimation can be treated as a supervised learning problem: train a neural network on hundreds of thousands of sample sets drawn from synthetic distributions whose MI is known, and let it learn to predict MI directly from data. The authors introduce MIST, a permutation-invariant attention model, and MISTQR, its quantile-regression variant. On test distributions with 10–500 samples, dimensions 2–32, and MI values up to 40, they report roughly 10x lower mean squared error than the strongest classical baseline (a k-nearest-neighbor estimator) on distributions seen in training, and about 5x lower on held-out distribution families. The quantile intervals are reported well-calibrated, inference is a single forward pass, and generalization holds for unseen families within the same synthetic generation geometry. The paper is explicit that these estimators carry no a priori consistency guarantees and depend on the diversity of the meta-training distribution.

What carries the argument

MIST is a SetTransformer++-based network with a two-dimensional attention scheme: one attention pass over samples (permutation-invariant, since samples have no intrinsic order) and one over dimensions (to handle variable input dimensionality). It is trained with mean squared error on 625,000 synthetic joint distributions with analytically known MI; MISTQR instead optimizes a pinball (quantile) loss so that querying different tau values yields quantiles of the sampling distribution. The paper also shows that the Bayes-optimal solution is the posterior expectation of MI given the dataset, and that the MSE decomposes into an approximation error plus an irreducible epistemic uncertainty term.

What would settle it

Evaluate MIST/MISTQR on distributions that do not arise from the same invertible-transformation pipeline—e.g., discrete/categorical data, high-mode mixtures, or time series—and measure bias and quantile coverage; or run the paper's own data-processing and additivity self-consistency tests with a strict tolerance. If the estimates move monotonically away from truth, or the DPI/additivity ratios deviate far from 1 across a broad sample of inputs, the learned function is not estimating mutual information but some calibrated proxy.

Watch

Extended reading notes

Core claim

The central claim is that an estimator parameterized as a neural network and trained end-to-end on a large meta-dataset of joint distributions with known ground-truth MI can outperform classical and neural MI estimators in the hard regime of small samples, moderate dimensions, and high MI values—including on distribution families never seen during training. The learned estimator avoids the systematic underestimation that plagues existing methods, and the quantile variant approximates the full sampling distribution of MI. The authors frame this as a deliberate paradigm shift: instead of deriving estimators from density or density-ratio theory, the estimator is learned, with the trade-off that

Load-bearing premise

The load-bearing premise is that the synthetic meta-distribution of joint laws—built by applying invertible transformations to a small set of base distributions—is representative enough of the distributions an MI estimator will encounter in deployment; the paper states that generalization to distant out-of-distribution families cannot be guaranteed a priori, and the 'unseen' test families come from the same synthetic generation pipeline.

Editorial extensions

If this is right

  • If the empirical results hold, MI estimation becomes an amortized operation: one training run, then every new dataset is a single forward pass, making large-scale feature selection and filtering practical.
  • The quantile-regression variant supplies calibrated confidence intervals without bootstrap refitting, at nearly the same cost as a point estimate.
  • Because the estimator is fully differentiable, it can be embedded as a module inside larger learning systems that optimize or regularize with MI.
  • The reported sample-efficiency advantage (roughly half the samples needed by the best classical baseline for the same accuracy) would extend MI-based analysis to regimes where data collection is expensive.
  • The paper's own self-consistency tests show the learned quantity does not automatically satisfy the data processing inequality or additivity; future versions can inject these as losses, but current use should validate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same supervised meta-training pipeline could be applied to other information-theoretic quantities—conditional MI, KL divergence, entropy—by generating matching meta-datasets; nothing in the framework is specific to MI itself.
  • Since MI is invariant under invertible transformations, normalizing flows could map the synthetic meta-distribution onto a target data modality; a concrete test would be whether a flow-adapted MIST beats a general MIST on real image or text embeddings.
  • The paper's ablation suggests that training on a mix of dimensions helps high-dimensional accuracy but requires more data; a natural extension is a curriculum that exposes the model to dimensions in increasing order.
  • The MSE decomposition gives a way to measure headroom: by estimating the irreducible variance term for a given meta-distribution, one could tell how much of a learned estimator's error is due to finite training versus the inherent difficulty of the inference problem.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MIST, a neural-network MI estimator trained on 625,000 synthetic joint distributions with known MI generated by the BMI library. The architecture is a SetTransformer variant with attention over samples and dimensions, yielding permutation-invariance and variable n,d. MIST is trained with MSE; MISTQR uses pinball loss to output quantiles. Evaluation is on BMI-derived IMD and OoMD splits at n=10-500, d=2-32, MI=0-40. The reported results show MIST/MISTQR roughly 10x lower MSE than KSG on IMD and 5x on OoMD, with faster inference; Section 5 explicitly disclaims consistency guarantees and documents failures on DPI/additivity self-consistency tests.

Significance. If the empirical claims are taken at face value, the paper makes a meaningful contribution: it provides a large, reproducible benchmark (code released) and demonstrates that supervised meta-training can produce fast, low-MSE MI estimators in the low-sample regime where KSG is the standard. The theoretical framing via the Bayes-optimal regression decomposition (Eq. 2) is sound, and the known ground-truth labels from BMI give a credible internal benchmark. However, the two strongest advertised conclusions - generalization to 'unseen distributions' and 'well-calibrated' quantile intervals - are materially weaker than the abstract and Section 4 suggest once Appendix B Table 4 and Appendix D.1 Table 8 are read carefully.

major comments (4)
  1. [Abstract; §4.5; Appendix B Table 4] The OoMD claim is weaker than stated. Table 4 shows that every OoMD family is a BMI structure/transform combination of the same base laws used in Mtrain (multi normal, multi student, multi additive noise). In particular, Mtrain already contains 'multi additive noise-wigglify', and OoMD tests 'multi additive noise-[base, halfcube]'. Thus 'unseen' means unseen combinations within the same BMI-generated manifold, not new data-generating processes. Since Eq. (2) shows the learned estimator converges to the posterior mean under P_Gamma, the 5x OoMD gain is evidence of interpolation inside the training meta-distribution. Section 5 correctly disclaims guarantees for distant OoMD families, but the abstract and Section 4.1 should be reworded to avoid claiming generalizable 'unseen distribution' performance.
  2. [§4.3; Appendix D.1 Table 8] The calibration claim is contradicted by the paper's own consistency analysis. Section 4.3 says MISTQR 'remains well-calibrated for unseen distributions', but Table 8 reports 15.8% lower-bound and 27.5% upper-bound failures on the OoMD split even when using the extreme quantiles tau=0 and tau=1. That means over a quarter of OoMD test points fall outside the predicted support of the sampling distribution. These numbers need to be reported in the main text and the calibration claim must be qualified; as written, the abstract's 'well-calibrated' is not supported.
  3. [§4.5; Fig. 7] The generalization study in Figure 7 shows a clear limitation that is omitted from the abstract: for unseen distributions at higher dimensionality than seen in training, MIST/MISTQR degrade substantially. The current claim of generalization 'to unseen distributions and sample sizes' (Section 4.5) is only true within the dimensions/sample sizes seen for OoMD data. The paper should either limit the claim to in-range generalization or report the Fig. 7 caveat prominently.
  4. [Appendix D.1; Section 5] Section 5 appropriately discloses that the estimators fail the data processing and additivity self-consistency tests and have no consistency guarantee, but this is a load-bearing limitation for practical use. Because the estimator has no n->infinity guarantee and no error bounds outside the meta-distribution, the manuscript should present the DPI/additivity failures in the main experiments or at least clearly tie them to the recommended usage conditions. This would prevent the 'fully data-driven' framing from being read as a replacement for classical estimators in settings where consistency is required.
minor comments (5)
  1. [Appendix D.1] Typo in bound-check definition: the second condition should compare I_true to \hat I_upper, not \hat I_lower. As written, 'upper bound failures' is undefined.
  2. [Appendix B Table 4 caption] The text says 'a reduced test subset of 1k samples is used for both IMD and OoMD splits', but Table 4 lists 1,080 and 1,260 points and Section 4.1 uses 2,340 in total. The numbers should be reconciled.
  3. [Appendix C.1] The paragraph says results are 'summarized in Table 10', but the displayed item is called Figure 10. The cross-reference is inconsistent.
  4. [§4.2 / Fig. 4] The MSE thresholds 0.03, 0.07, and 0.09 are chosen 'based on prior heatmap analysis'; their substantive meaning for practitioners is not explained. A sentence connecting thresholds to Table 1 MSE values would help.
  5. [§3] The sentence 'the model must process datasets of varying size while remaining invariant to sample ordering' might be clearer as 'permutation equivariant in the sample dimension before pooling'; the current wording is fine but could confuse readers.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor near-by-construction calibration claim; central MSE outperformance is a genuinely held-out measurement, so overall circularity is low.

  1. fitted input called prediction [Section 3 (Eq. 3 pinball loss) vs Section 4.3 / Figure 5 calibration claim]
    "we also train a quantile regression model (MISTQR) to predict the τ-quantile of the sampling distribution for any given τ∈[0,1] using the pinball loss ... The quantile-loss variant MISTQR achieves nearly perfect calibration on distributions seen in training and remains well-calibrated for unseen distributions."

    Minimizing the pinball loss (Eq. 3) over the training meta-distribution P_Gamma has as its optimum the conditional τ-quantile of I_gamma given D. The IMD half of the calibration check in Section 4.3 / Fig. 5 measures, on held-out draws from the same distribution families that generated the training data, exactly the property this loss optimizes: how often the true MI falls below the predicted quantile. 'Nearly perfect calibration on distributions seen in training' is therefore largely a restatement of the fitted objective (a check that optimization converged), not an independent validation. The out-of-distribution part of the claim is weakened by the paper's own Appendix D.1 Table 8 (15.8% lower-bound and 27.5% upper-bound failures on OoMD) and by the Section 5 disclaimer. The central Tabl

full rationale

The derivation chain is essentially self-contained. Ground-truth MI labels are computed analytically for base distributions and preserved by BMI's invertible transformations (Appendix B), so the supervised target is independent of the estimator being trained; the model never supplies its own labels. Equations (1)-(2) are standard MSE-regression facts (Bayes-optimal predictor is the posterior mean, with an irreducible variance term), and Eq. (3) is standard quantile regression; neither imports a theorem from the authors' prior work. The headline results (Table 1: MIST ~3.1e0 vs KSG ~3.0e1 on IMD; ~8e0 vs ~3.5e1 on OoMD) are external measurements against analytical labels on held-out family combinations, not quantities forced by construction. Two caveats keep the score above zero while remaining sub-circular. First, MISTQR's near-perfect IMD calibration is close to tautological with its own training loss, since pinball-loss minimization targets the conditional quantile under the training meta-distribution; the OoMD part is genuinely out-of-distribution and is partially contradicted by the paper's own Appendix D.1 failure rates. Second, the OoMD 'unseen distributions' are unseen combinations of the same base laws (multinormal, student-t, additive noise) and the same BMI transformation library used in training (Table 4), so the ~5x OoMD gain is interpolation within the BMI-generated manifold rather than evidence of broad out-of-distribution generalization; the paper explicitly discloses this in Section 5 ('generalization to distant out-of-distribution families cannot be guaranteed a priori'), and Section 4.5 shows degradation on OoMD data with unseen dimensions. The self-citation to Peyrard & Cho (2025), whose authors overlap with the present paper, situates the meta-statistical paradigm but is not load-bearing: the convergence argument and the empirical comparisons stand independently. Overall, no load-bearing step reduces by definition to its inputs; the flagged items are an external-validity scope limit and one secondary, near-by-construction calibration claim.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No new physical entities are introduced. The main external inputs are the hand-chosen meta-distribution P_Gamma and BMI's ground-truth MI labels; both are entered as axioms. Neural network weights are standard fitted parameters rather than ad hoc free quantities.

free parameters (2)
  • lambda_phys = 0.3
    Regularization weight in the ablated four-coefficient entropy model (Appendix C.1, Eq. 9). Not used in the final direct-MI model, but an ad hoc hand-set constant in the paper.
  • MSE thresholds = 0.03, 0.07, 0.09
    Sample-size requirement thresholds in Figure 4, stated as 'selected based on our prior heatmap analysis' — post hoc, not used to fit the model.
assumptions (5)
  • standard math Mutual information is invariant under invertible transformations of the variables.
    Used to generate known-MI training data (Appendix B) and proposed for normalizing-flow domain adaptation (Section 5).
  • domain assumption The BMI library computes the ground-truth MI of every generated distribution correctly, analytically or numerically.
    All 625k training labels and the test labels rely on this computation; an error in BMI would propagate to every estimate and comparison.
  • domain assumption The meta-distribution P_Gamma over training distributions is representative of deployment distributions.
    The estimator minimizes MSE under training P_Gamma; if target distributions differ, no guarantees exist. Acknowledged in Section 5.
  • domain assumption The SetTransformer++ architecture has sufficient capacity and is trained well enough to approximate the Bayes-optimal regression function f*(D) = E[I_gamma | D].
    The MSE decomposition in Eq. 2 assumes the square loss and an optimal predictor; capacity and optimization are not formally verified.
  • domain assumption Quantile predictions under the training meta-distribution describe the 'sampling distribution' of MI for a held-out dataset.
    Quantile intervals are posterior predictive intervals under the meta-distribution, not frequentist sampling distributions; calibration on held-out BMI data does not imply general calibration.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MIST: Mutual Information Estimation Via Supervised Training." pith.science (2026). https://pith.science/paper/2YAJ7I5T

@misc{pith2026251118945,
  author       = {Pith},
  title        = {Pith review of: MIST: Mutual Information Estimation Via Supervised Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2YAJ7I5T}},
  note         = {Machine review of arXiv:2511.18945}
}
read the original abstract

We propose a fully data-driven approach to designing mutual information (MI) estimators. Since any MI estimator is a function of the observed sample from two random variables, we parameterize this function with a neural network (MIST) and train it end-to-end to predict MI values. Training is performed on a large meta-dataset of 625,000 synthetic joint distributions with known ground-truth MI. To handle variable sample sizes and dimensions, we employ a two-dimensional attention scheme ensuring permutation invariance across input samples. To quantify uncertainty, we optimize a quantile regression loss, enabling the estimator to approximate the sampling distribution of MI rather than return a single point estimate. This research program departs from prior work by taking a fully empirical route, trading universal theoretical guarantees for flexibility and efficiency. Empirically, the learned estimators largely outperform classical baselines across sample sizes and dimensions, including on joint distributions unseen during training. The resulting quantile-based intervals are well-calibrated and more reliable than bootstrap-based confidence intervals, while inference is orders of magnitude faster than existing neural baselines. Beyond immediate empirical gains, this framework yields trainable, fully differentiable estimators that can be embedded into larger learning pipelines. Moreover, exploiting MI's invariance to invertible transformations, meta-datasets can be adapted to arbitrary data modalities via normalizing flows, enabling flexible training for diverse target meta-distributions.

Figures

Figures reproduced from arXiv: 2511.18945 by the authors.

Figure 1
Figure 1. We propose a fully data-driven, empirical approach to designing mutual information esti [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Heatmaps providing a detailed analysis of the performance of the two proposed MI estima [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Predicted MI as a function of the true MI [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: The MIST and MISTQR models scale substantially better to higher dimensions, requiring roughly half as many samples as KSG to achieve an MSE below the selected thresholds. The “+” marker indicates that over 500 samples (nrow) are required to obtain accurate estimates in…
Figure 5
Figure 5. Figure 5: Calibration of the proposed models on Mtest,IMD (left) and Mtest,OoMD (right), computed directly from quantiles for MISTQR and through bootstrap resampling for MIST and KSG. We report the mean absolute error in the legend [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Inference time for each method as a function of the sample size. Results are aver￾aged over 100 inferences (5 for MINDE), each performed over 10 meta-datapoints. The previous sections demonstrate that the learned algorithm generalizes effectively to un￾seen distributio…
Figure 7
Figure 7. Figure 7: Generalization performance (MSE) of models trained with limited dimensionality and [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Average MSE obtained by variable- and fixed-dimensionality versions of MIST on subsets [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 10
Figure 10. Figure 10: Mean MI estimates across prediction strategies [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Average MSE obtained by variable- and fixed-dimensionality versions of MIST on sub [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Average MSE obtained by variable- and fixed-dimensionality versions of MIST [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Average MSE obtained by variable- and fixed-dimensionality versions of MIST [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: Average MSE of the proposed models with variable input dimensionality versus [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: Average bias magnitude of the proposed models with variable input dimensionality versus [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]
Figure 16
Figure 16. Figure 16: Average variance of the proposed models with variable input dimensionality versus [PITH_FULL_IMAGE:figures/full_fig_p027_16.png]
Figure 17
Figure 17. Figure 17: Detailed visualization of the average MSE values for our two proposed models across [PITH_FULL_IMAGE:figures/full_fig_p029_17.png]
Figure 18
Figure 18. Figure 18: Independence Test: A is an MNIST image, and Br contains the top r rows of image A. For a consistent estimator, the plotted MI ratio will be non-decreasing and approach 1 with an increasing number of shared rows. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_18.png]
Figure 19
Figure 19. Figure 19: Data Processing Test: A is an MNIST image, and Br contains the top r rows of image A. This test compares I([A, A], [Br, B′r−3 ′ ]) to I([A, Br]), where A is an original MNIST image, Br is the first r rows of A, and so B′r−3 ′ is a further masked version of image A. Th…
Figure 20
Figure 20. Figure 20: Additivity Test: A is an MNIST image, and Br contain the top r rows of image A. The premise of this test is that adding a second independent sample should double the mutual information estimate. The ratio compares I([A1, A2], [B1r, B2r]) to I([A, Br]), and should be n…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

5 extracted references · 4 linked inside Pith

  1. [2017]

    Lily Zhang, Veronica Tozzo, John Higgins, and Rajesh Ranganath

    URLhttps://proceedings.neurips.cc/paper_files/paper/2017/ file/f22e4747da1aa27e363d86d40ff442fe-Paper.pdf. Lily Zhang, Veronica Tozzo, John Higgins, and Rajesh Ranganath. Set norm and equivariant skip connections: Putting the deep in deep sets. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.),Proceeding...

    arXiv 2017

  2. [2021]

    Jiaming Song and Stefano Ermon

    URLhttps://proceedings.neurips.cc/paper_files/paper/2021/ file/56c3b2c6ea3a83aaeeff35eeb45d700d-Paper.pdf. Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators.arXiv preprint arXiv:1910.06222, 2019. Ingo Steinwart and Andreas Christmann. Estimating conditional quantiles with the help of the pinball los...

    arXiv 2021

  3. [2022]

    Samuel M ¨uller, Matthias Feurer, Noah Hollmann, and Frank Hutter

    URLhttps://openreview.net/forum?id=KSugKcbNf9. Samuel M ¨uller, Matthias Feurer, Noah Hollmann, and Frank Hutter. PFNs4BO: In-context learn- ing for Bayesian optimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.),Proceedings of the 40th International Conference on Machine Learning, vo...

    arXiv 2023

  4. [2318]

    Samuel M¨uller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter

    URLhttps://link.aps.org/doi/10.1103/PhysRevE.52.2318. Samuel M¨uller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Trans- formers can do bayesian inference. InInternational Conference on Learning Representations,

    doi:10.1103/physreve.52.2318

  5. [4454]

    Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh

    PMLR, 2020. Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes.arXiv preprint arXiv:1901.05761, 2019. Allon M Klein, Linas Mazutis, Ilke Akartuna, Naren Tallapragada, Adrian Veres, Victor Li, Leonid Peshkin, David A Weitz, and Marc W Kirschner. Droplet barcoding...

    arXiv 2020

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.