Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Vendi Information Gain: An Alternative To Mutual Information For Science And Machine Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes Vendi Information Gain, a sample-based, similarity-aware information gain measure that reduces to mutual information when samples are completely dissimilar, and shows it remains stable where MI fails.

desk verdict A useful sample-based information measure with a correct q=1 MI-recovery proof, but the unqualified boundedness claim is false for q>1 and the paper needs code and kernel sensitivity analysis. read the letter →

arxiv 2505.09007 v2 pith:JIIDIA44 submitted 2025-05-13 cs.IT math.IT

Quan Nguyen , Adji Bousso Dieng This is my paper · ORCID
classification cs.ITmath.IT MSC 94A17
keywords VendiInformationGainmutualScoresimilaritykernelactivedataacquisitionlevel-setestimationRényientropy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that information gain can be quantified by Vendi Information Gain (VIG): the expected drop in Vendi entropy—the Rényi entropy of the eigenvalues of a similarity kernel matrix—of a variable after observing another. Unlike Shannon's mutual information, VIG is computed directly from samples, requires no tractable density, is asymmetric, and incorporates pairwise similarity between sample values. VIG reduces exactly to mutual information when all samples are treated as completely dissimilar and the order is $q=1$, so MI is a special case. If correct, VIG would provide an information-theoretic criterion that remains usable where MI is intractable or misleading, including active data acquisition and level-set estimation.

What carries the argument

The key object is the Vendi score and its logarithm, the Vendi entropy, defined as the Rényi entropy of the normalized eigenvalues of a positive semidefinite similarity kernel matrix $K$ built from samples. VIG is the expected reduction in this Vendi entropy after conditioning on another variable. The kernel is what carries the argument: it encodes which samples count as similar, so the resulting information gain respects distances rather than treating every distinct outcome as equally separated. The identity kernel collapses Vendi entropy to ordinary Rényi entropy of the sample probabilities, which is exactly the step that makes VIG recover mutual information at $q=1$.

What would settle it

Compute VIG for a fixed two-class data set under a Gaussian kernel with two very different bandwidths, one much smaller than the typical inter-sample distance and one comparable to the class separation; if the ordering of candidate queries or channels reverses, the claim that VIG reliably captures information gain fails. Equivalently, a single finite-sample example where independent variables give $\mathrm{VIG} < 0$ would contradict the paper's boundedness claim.

Watch

Extended reading notes

Core claim

The paper defines Vendi information gain as $\mathrm{VIG}(\theta, y; q) = H_V(\mathcal{D}; q) - \mathbb{E}_y[H_V(\mathcal{D}_y; q)]$, where $H_V$ is Vendi entropy, the Rényi entropy of normalized eigenvalues of a kernel matrix on samples. The central discovery is that this quantity is a valid, directional information-gain measure that accounts for similarity: when the kernel matrix is the identity, Vendi entropy becomes Rényi entropy, and at order $q=1$ VIG equals $I(\theta; y)$. The paper further claims that VIG is zero for independent variables, is bounded between zero and the initial Vendi entropy, is additive under independent variables, and is sensitive to the spread of the variable's distribution; it also demonstrates that VIG estimates remain stable where MI estimates become biased or degenerate.

Load-bearing premise

The entire measure depends on a user-chosen similarity kernel and its hyperparameters; the paper acknowledges this choice is fully up to the user, and if rankings of information gain change under reasonable kernel choices, VIG's advantage over MI is not intrinsic.

Editorial extensions

If this is right

  • Any pipeline that currently approximates MI from samples in high dimensions can instead use VIG without density estimation, and the paper's experiments show VIG estimates do not collapse to zero as sample size shrinks.
  • Because VIG is asymmetric, it can model directional information flow in communication, causal, and cognitive settings where MI's symmetry is a known limitation.
  • VIG's additivity under independence lets a joint system's Vendi entropy decompose into a sum of individual entropies, so computing VIG can be reduced to smaller eigendecompositions.
  • VIG yields a concrete, sample-only active data acquisition and level-set estimation policy; in the paper's experiments it consistently achieves higher F1 scores than MI, STRADDLE, LSE, and uncertainty sampling.
  • Setting $q=1$ and the identity kernel recovers MI, so VIG is a strict generalization rather than a competing ad hoc quantity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit: because VIG is asymmetric, comparing $\mathrm{VIG}(\theta; y)$ with $\mathrm{VIG}(y; \theta)$ could serve as a directional dependence diagnostic in causal or directed-information settings, though the paper only gestures at this.
  • The paper reports no sensitivity analysis over kernel bandwidth; a natural check is whether the channel-ordering and active-learning rankings shown in its figures survive small perturbations of the Gaussian lengthscale. If rankings flip, the practical advantage over MI is kernel-tuned rather than structural.
  • Because VIG is computed entirely from posterior samples, it could be plugged into any sampling-based acquisition scheme, including those using neural-network posteriors or implicit level-set thresholds; the paper mentions such connections but does not test them.
  • A stronger validation would be on synthetic problems where the true information gain is known in closed form: fixing a kernel, one could compare VIG's ranking of candidate queries against the exact MI ranking to see whether similarity-sensitivity always improves the measure or only helps in multimodal regimes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes the Vendi Information Gain (VIG), defined in Eq. (4) as the expected reduction in Vendi entropy, VIG(θ,y;q) = H_V(D;q) − E_y[H_V(D_y;q)], where H_V is the Rényi entropy of the normalized eigenvalues of a (possibly probability-weighted) similarity kernel matrix. The authors claim that VIG generalizes mutual information, recovers MI for the identity kernel at q=1, is bounded between 0 and the initial entropy, is asymmetric, and is additive under independence. They illustrate these properties on communication channels and in three application areas: modeling human response times, active data acquisition (step function learning, death processes, location sensing), and level-set estimation with Gaussian-process surrogates. Main experiments use q=1, while Appendix B reports ablations over q.

Significance. If the theoretical claims are corrected, VIG is a useful sample-based, similarity-aware alternative to mutual information. The derivation that VIG equals MI for the identity kernel at q=1 is clean and parameter-free, and the level-set estimation experiments show promising practical potential. The paper does not ship code at review time, and the response-time simulation sets similarity values by hand, so the empirical component is a demonstration rather than a full validation. The main weaknesses are mathematical: the boundedness claim is false as stated for q>1, and the additivity proof is incomplete. These issues are load-bearing for the characterization of VIG as a valid information-gain measure and need to be fixed before the central claims are accepted.

major comments (4)
  1. [Section 3.2, Boundedness] The claim that VIG ranges between 0 and the initial entropy is asserted without proof and is false for q>1, even under the identity kernel where the probability-weighted Vendi entropy of Eqs. (2)-(3) reduces to the Rényi entropy of the distribution. Concretely, take θ with three outcomes and Y binary with P(Y=0)=P(Y=1)=1/2, P(θ|Y=0)=(0.936,0.03,0.034), P(θ|Y=1)=(0.864,0.07,0.066), and marginal P(θ)=(0.9,0.05,0.05). With K=I and q=2, H_2(θ)=−ln(0.815)≈0.204, while E_Y[H_2(θ|Y)] = 0.5[−ln(0.878152)−ln(0.755752)]≈0.205, so VIG≈−7×10^{-4}<0. Since q is a free hyperparameter, including the ablation values q=0.1,0.2,2,∞ in Appendix B, the unqualified boundedness claim, and the associated statement that VIG is a valid information-gain measure, are false as stated. The authors should restrict the claim to q=1, prove sufficient conditions on q and the kernel for nonnegativity, or qualify the claim explicitly.
  2. [Section 3.2, Additivity under independence, Eqs. (8)-(12)] Equation (8) states that under conditional independence VIG((x,y),z) = VIG(x,z) + VIG(y,z), but the proof in Eqs. (9)-(12) only establishes H_V(D_{x,y};q) = H_V(D_x;q) + H_V(D_y;q) for unconditional independence. The required conditional statement H_V(D_{x,y|z};q) = H_V(D_{x|z};q) + H_V(D_{y|z};q) is never shown. In addition, the decomposition K_{x,y}=K_x⊗K_y in Eq. (10) is an assumption about the joint kernel, not a consequence of independence; it must be stated explicitly as a product-kernel assumption. As written, Eq. (8) is unproved.
  3. [Section 4 and Figure 1] The paper claims that VIG estimates are stable where MI estimates fail, but it provides no finite-sample statistical characterization of the plug-in estimator \widehat{VIG} = \widehat{H}_V(D) − E_y[\widehat{H}_V(D_y)]. No bias, variance, consistency, or sample-complexity result is stated, and no conditions are given on the kernel bandwidth or the number of samples. Given that the paper explicitly motivates VIG as a sample-based quantity, a theoretical or at least a systematic empirical characterization of the estimator is needed to support the claim that VIG 'does not degenerate to 0 with decreasing samples like MI'.
  4. [Section 6 and Section 4] The paper acknowledges in Section 6 that kernel choice depends on the user, but none of the experiments reports a sensitivity analysis over the Gaussian kernel bandwidth or compares alternative kernels. Because VIG's values and rankings depend directly on the kernel, the claimed advantages over MI in the applications (Figures 2, 4, 5) may not be robust to this choice. A sensitivity analysis over kernel hyperparameters is needed to substantiate the general claim that VIG is a reliable information-gain measure.
minor comments (5)
  1. [Section A.2] The text says 'the identify kernel'; this should be 'the identity kernel'.
  2. [Section 3.2, Reducing to 0 with independent variables] The statement that VIG equals 0 when θ does not depend on y holds only at the population level or in expectation; for finite samples, the empirical VIG will generally not be exactly zero even under independence. The text should state this qualification.
  3. [Section 4.1 and Eq. (23)] The response-time simulation sets the similarity parameter s by hand (s=0, 0.5, 0.9) and reports agreement with Slamecka (1963); because s is not estimated from data, the match is a demonstration rather than a validation. This should be stated explicitly in the main text, not only in Appendix A.
  4. [Eq. (2)] There is a typographical error in the definition of the probability-weighted kernel matrix: 'diag(pp)' should be 'diag(p)'.
  5. [Section A.4 and Figure 4] The appendix says each run is repeated 10 times, while the caption of Figure 4 reports averages across 50 repeats. Please clarify which number is correct.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: VIG's definition, the MI-recovery proof, and the main benchmarks are self-contained; the false 'Boundedness' claim is a correctness defect, not a circularity defect.

full rationale

The central derivation is not circular. VIG is defined in Eq. (4) as H_V(D;q) - E_y[H_V(D_y;q)], a new quantity built from the Vendi entropy of Eq. (3); it is not fitted to any target quantity. The recovery of MI is a direct mathematical consequence: Eqs. (5)-(6) take the kernel to be the identity, so the probability-weighted kernel matrix is diag(p), its eigenvalues are p, the q=1 Vendi entropy is Shannon entropy, and VIG(θ,y;q=1)=H(θ)-H(θ|y)=I(θ;y). No hidden input is being re-labeled as a prediction. The experimental sections compare VIG with MI and other baselines on external or simulated benchmarks (GP synthetic LSE tasks, Andrade-Pacheco et al. (2020) disease data, step/death/location-sensing tasks), and the q-ablation in Appendix B tests robustness rather than fitting. The response-time simulation chooses kernel similarities s∈{0,0.5,0.9} to illustrate a qualitative regime (Eq. (23)); this is a demonstration of the measure's behavior, not a parameter fit to the Slamecka (1963) data, so no fitted-input-called-prediction step is present. Self-citations to Friedman & Dieng (2023) and Pasarkar & Dieng (2024) supply the Vendi-score background, but the load-bearing 'generalizes MI' step is proven within this paper, and the benchmarks provide independent evidence. Separately, Section 3.2 'Boundedness' asserts without proof that 'VIG ranges between 0 minimally (when θ does not depend on y, and D ≡ D_y, ∀y) and the initial entropy maximally (when y completely determines θ, and D_y contains one member)'; this is not established, and for q>1 a discrete counterexample with identity kernel already gives negative VIG, so the boundedness claim is false as stated. That is a correctness/validity defect, not a circularity defect, and does not raise the circularity score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central construction of VIG relies on the user-provided kernel and the Vendi entropy. No new physical or mathematical entities are introduced. The main unstated assumptions are the nonnegativity of VIG and the adequacy of the Monte Carlo approximations, both of which are load-bearing for the experimental claims.

free parameters (3)
  • Vendi order q = q = 1 in main experiments
    Order of the Rényi entropy; it is user-chosen. Appendix B ablates q and shows robustness, but the main results use q = 1.
  • Kernel choice and hyperparameters (Gaussian bandwidth, Hamming kernel) = Not reported for most experiments
    All VIG values depend on the similarity kernel. No sensitivity analysis is given, so the reported results may depend on arbitrary kernel choices.
  • Similarity values s in response-time simulation = s = 0, 0.5, 0.9
    Chosen by hand to represent low, intermediate, and high similarity; used to produce the trend matching Slamecka (1963), not fitted to data.
assumptions (4)
  • standard math Kernel matrix K is positive semidefinite with k(θ,θ)=1 and is normalized to unit trace.
    Required by the Vendi Score definition in Section 2, eq. 1.
  • domain assumption Samples D and D_y are i.i.d. from p(θ) and p(θ|y) respectively, and the Monte Carlo expectation in eq. 4 is a good approximation.
    All experiments use Monte Carlo estimates; no convergence or bias analysis is provided.
  • ad hoc to paper Conditioning reduces Vendi entropy on average, so VIG ≥ 0.
    Boundedness is asserted in Section 3.2 without proof; finite-sample counterexamples may exist because a subsample can have higher Vendi entropy than the full sample.
  • ad hoc to paper Kernel hyperparameters are fixed and do not qualitatively change comparisons.
    No sensitivity analysis over kernel bandwidth is provided; the paper states kernel choice is user-dependent in Section 6.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Vendi Information Gain: An Alternative To Mutual Information For Science And Machine Learning." pith.science (2026). https://pith.science/paper/JIIDIA44

@misc{pith2026250509007,
  author       = {Pith},
  title        = {Pith review of: Vendi Information Gain: An Alternative To Mutual Information For Science And Machine Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JIIDIA44}},
  note         = {Machine review of arXiv:2505.09007}
}
read the original abstract

In his 1948 seminal paper A Mathematical Theory of Communication that birthed information theory, Claude Shannon introduced mutual information (MI), which he called "rate of transmission", as a way to quantify information gain (IG) and defined it as the difference between the marginal and conditional entropy of a random variable. While MI has become a standard tool in science and engineering, it has several shortcomings. First, MI is often intractable - it requires a density over samples with tractable Shannon entropy - and existing techniques for approximating it often fail, especially in high dimensions. Moreover, in settings where MI is tractable, its symmetry and insensitivity to sample similarity are undesirable. In this paper, we propose the Vendi Information Gain (VIG), a novel alternative to MI that leverages the Vendi scores, a flexible family of similarity-based diversity metrics. We call the logarithm of the VS the Vendi entropy and define VIG as the difference between the marginal and conditional Vendi entropy of a variable. Being based on the VS, VIG accounts for similarity. Furthermore, VIG generalizes MI and recovers it under the assumption that the samples are completely dissimilar. Importantly, VIG only requires samples and not a probability distribution over them. Finally, it is asymmetric, a desideratum for a good measure of IG that MI fails to meet. VIG extends information theory to settings where MI completely fails. For example, we use VIG to describe a novel, unified framework for active data acquisition, a popular paradigm of modern data-driven science. We demonstrate the advantages of VIG over MI in diverse applications, including in cognitive science to model human response times to external stimuli and in epidemiology to learn epidemic processes and identify disease hotspots in different countries via level-set estimation.

Figures

Figures reproduced from arXiv: 2505.09007 by the authors.

Figure 1
Figure 1. Illustration of MI’s failure modes and VIG’s benefits. MI estimates tend to become biased in unpredictable ways as the dimensionality increases in the second column and degenerate to 0 with fewer samples in the third column, while VIG estimates are relatively stable in both. Last column: Compared to MI, VIG better corresponds to the trend of the average predictive error conditioned on the label. limited samples (Möl… view at source ↗
Figure 2
Figure 2. Simulated response times as a function of similarity between possible messages by MI and VIG against observed experimental data from Slamecka (1963). MI’s insensitivity to inter-message similarity prevents the simulation of different behaviors under different conditions, while VIG’s simulations closely match observed real-world data. However, this model has its own limitations. As MI treats messages as completely di… view at source ↗
Figure 3
Figure 3. Illustration of Vendi information gain for active data acquisition. For each candidate x we may query, we first draw Monte Carlo samples of its label { yi } according to our predictive model trained on observed data fD. Conditioned on each sample, we then draw fantasized samples of all labels within the search space ξi , which are transformed by function g(·) to yield samples of the quantity of interest θi . Informa… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Average estimation error (± 1 standard error) across 50 repeats. VIG is competitive against MI, at times outperforming MI by a large margin. the posterior and the fixed prior entropy. In many cases, the posterior distribution p(θ | D) is not available in closed form, a…
Figure 5
Figure 5. Figure 5: Average F1 scores and standard errors achieved by LSE policies as a function of the number of queries. MI performs the worst, while VIG consistently achieves the highest F1 scores, sometimes significantly outperforming baselines. show this F1 score by each policy under…
Figure 6
Figure 6. Figure 6: Behavior comparison between different policies in one illustrative run under the Schistosomiasis in Malawi experiments. MI wastes many queries on easy-to-classify regions, while VIG focuses on multiple disease hotspots where classification is challenging, leading to a …
Figure 7
Figure 7. Figure 7: Progression of posterior probabilities and information gain, as measured by MI and VIG, under varying degrees of inter-message similarity as a function of time. MI plots the same trend across different conditions, while VIG becomes lower under more similar messages, ap…
Figure 8
Figure 8. Figure 8: Simulated response times as a function of similarity between possible messages by VIG of different orders q. 0 2 4 no. experiments 0.0 0.1 0.2 error step function 0 10 20 no. experiments 2 4 death process q = 0.1 q = 0.5 q = 1 q = 2 q = 0 10 20 no. experiments 0.2 0.5 …
Figure 9
Figure 9. Figure 9 [PITH_FULL_IMAGE:figures/full_fig_p030_9.png]
Figure 10
Figure 10. Figure 10: F1 scores achieved by VIG of different orders q as a function of the number of queries. VIG’s performance is mostly robust against the value of q. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. How Much Is a Dataset Worth? Scaling Laws, the Vendi Score, and Matrix Spectral Functions

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Vendi Score and scaling-law objectives belong to the class of matrix spectral functions, which are submodular, enabling efficient greedy selection of training data that outperforms random subsets in predicting held-ou...

Reference graph

Works this paper leans on

84 extracted references · 65 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  2. [2]

    and Michel, O

    Amblard, P.-O. and Michel, O. J. J. (2011). On directed information theory and granger causality graphs. Journal of Computational Neuroscience , 30(1):7--16

    2011

  3. [3]

    F., van der Laan, M

    Andrade-Pacheco, R., Rerolle, F., Lemoine, J., Hernandez, L., Me \" t \'e , A., Juziwelo, L., Bibaut, A. F., van der Laan, M. J., Arnold, B. F., and Sturrock, H. J. (2020). Finding hotspots: development of an adaptive spatial sampling approach. Scientific Reports , 10

    2020

  4. [4]

    and Polani, D

    Ay, N. and Polani, D. (2008). Information Flows in Causal Networks . Advances in Complex Systems , 11(01):17--41

    2008

  5. [5]

    Berns, S., Colton, S., and Guckelsberger, C. (2023). Towards Mode Balancing of Generative Models via Diversity Weights . arXiv preprint . arXiv:2304.11961 [cs.LG]

    arXiv 2023

  6. [6]

    C., Genovese, C

    Bryan, B., Nichol, R. C., Genovese, C. R., Schneider, J., Miller, C. J., and Wasserman, L. (2005). Active Learning For Identifying Function Threshold Boundaries . In Advances in Neural Information Processing Systems , volume 18

    work page 2005

  7. [7]

    S., and Gibbons, S

    Carr, A., Diener, C., Baliga, N. S., and Gibbons, S. M. (2019). Use and abuse of correlation analyses in microbial ecology. The ISME journal , 13(11):2647--2655

    work page 2019

  8. [8]

    and Vlachos, A

    Cheng, J. and Vlachos, A. (2024). Measuring Uncertainty in Neural Machine Translation with Similarity-Sensitive Entropy . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics , pages 2115--2128

    work page 2024

Show all 84 references
  1. [9]

    T., Johnson, H

    Christie, S. T., Johnson, H. R., and Schrater, P. R. (2023). Information-Theoretic Neural Decoding Reproduces Several Laws of Human Behavior . Open Mind , 7:675--690

    2023

  2. [10]

    R., Gibson, G

    Cook, A. R., Gibson, G. J., and Gilligan, C. A. (2008). Optimal Observation Times in Experimental Epidemic Processes . Biometrics , 64(3):860--868

    2008

  3. [11]

    Cover, T. M. (1999). Elements of Information Theory . John Wiley & Sons

    1999

  4. [12]

    Czy \.z , P., Grabowski, F., Vogt, J., Beerenwinkel, N., and Marx, A. (2023). Beyond Normal: On the Evaluation of Mutual Information Estimators . Advances in Neural Information Processing Systems , 36:16957--16990

    2023

  5. [13]

    Dayan, P., Kakade, S., and Montague, P. R. (2000). Learning and selective attention. Nature Neuroscience , 3(11):1218--1223

    2000

  6. [14]

    DeGroot, M. H. (1962). Uncertainty, information, and sequential experiments. The Annals of Mathematical Statistics , 33(2):404--419

    1962

  7. [15]

    Dember, W. N. (1957). The relation of decision-time to stimulus similarity. Journal of Experimental Psychology , 53(1):68

    1957

  8. [16]

    Dorval, A. D. (2011). Estimating Neuronal Information: Logarithmic Binning of Neuronal Inter-Spike Intervals . Entropy , 13(2):485--501

    2011

  9. [17]

    L., and Engle, R

    Draheim, C., Hicks, K. L., and Engle, R. W. (2016). Combining Reaction Time and Accuracy: The Relationship Between Working Memory Capacity and Task Switching as a Case Example . Perspectives on Psychological Science , 11(1):133--155

    2016

  10. [18]

    Dushenko, S., Ambal, K., and McMichael, R. D. (2020). Sequential Bayesian Experiment Design for Optically Detected Magnetic Resonance of Nitrogen-Vacancy Centers . Physical Review Applied , 14(5):054036

    2020

  11. [19]

    R., Malik, I., and Rainforth, T

    Foster, A., Ivanova, D. R., Malik, I., and Rainforth, T. (2021). Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Design . In International Conference on Machine Learning , pages 3384--3395

    2021

  12. [20]

    and Dieng, A

    Friedman, D. and Dieng, A. B. (2023). The Vendi Score: A Diversity Evaluation Metric for Machine Learning . Transactions on Machine Learning Research

    2023

  13. [21]

    Q., Barbour, D., and Cunningham, J

    Gardner, J., Malkomes, G., Garnett, R., Weinberger, K. Q., Barbour, D., and Cunningham, J. P. (2015). Bayesian active model selection with an application to automated audiometry. Advances in Neural Information Processing Systems , 28

    2015

  14. [22]

    Gotovos, A., Casati, N., Hitz, G., and Krause, A. (2013). Active Learning for Level Set Estimation . In The International Joint Conference on Artificial Intelligence , pages 1344--1350

    2013

  15. [23]

    Ha, H., Gupta, S., Rana, S., and Venkatesh, S. (2021). High Dimensional Level Set Estimation with Bayesian Neural Network . In The AAAI Conference on Artificial Intelligence , volume 35, pages 12095--12103

    2021

  16. [24]

    u r die Medizin 2005: Algorithmen—Systeme—Anwendungen Proceedings des Workshops vom 13.--15. M \

    Haber, E. and Modersitzki, J. (2005). Beyond mutual information: A simple and robust alternative. In Bildverarbeitung f \"u r die Medizin 2005: Algorithmen—Systeme—Anwendungen Proceedings des Workshops vom 13.--15. M \"a rz 2005 in Heidelberg , pages 350--354. Springer

    2005

  17. [25]

    Hafemann, P., Song, X., and Brecht, D. (2024). A Novel Information-Theoretic Metric for Evaluating LiDAR Setups of Autonomous Vehicles . In IEEE International Conference on Systems, Man, and Cybernetics (SMC) , pages 1270--1277. IEEE

    2024

  18. [26]

    J., Ross, C., Williams, A., Drozdzal, M., and Soriano, A

    Hall, M., Bell, S. J., Ross, C., Williams, A., Drozdzal, M., and Soriano, A. R. (2024). Towards Geographic Inclusion in the Evaluation of Text-to-Image Models . In ACM Conference on Fairness, Accountability, and Transparency , pages 585--601

    2024

  19. [27]

    J., Green, S

    Hart, S. J., Green, S. R., Casp, M., and Belger, A. (2010). Emotional priming effects during stroop task performance. Neuroimage , 49(3):2662--2670

    2010

  20. [28]

    Heitz, R. P. (2014). The speed-accuracy tradeoff: history, physiology, methodology, and behavior. Frontiers in Neuroscience , 8:150

    2014

  21. [29]

    Houlsby, N., Husz \'a r, F., Ghahramani, Z., and Lengyel, M. (2011). Bayesian Active Learning for Classification and Preference Learning . arXiv preprint . arXiv:1112.5745

    2011 arXiv

  22. [30]

    Hyman, R. (1953). Stimulus information as a determinant of reaction time. Journal of Experimental Psychology , 45(3):188

    1953

  23. [31]

    Jalali, M., Ospanov, A., Gohari, A., and Farnia, F. (2024). Conditional Vendi Score: An Information-Theoretic Approach to Diversity Evaluation of Prompt-based Generative Models . arXiv preprint . arXiv:2411.02817

    2024 arXiv

  24. [32]

    Kala, Z. (2021). Global Sensitivity Analysis Based on Entropy: From Differential Entropy to Alternative Measures . Entropy , 23(6):778

    2021

  25. [33]

    and Pilpel, Y

    Karmon, A. and Pilpel, Y. (2016). Biological causal links on physiological and evolutionary time scales. Elife , 5:e14424

    2016

  26. [34]

    Kinney, J. B. and Atwal, G. S. (2014). Equitability, mutual information, and the maximal information coefficient. Proceedings of the National Academy of Sciences , 111(9):3354--3359

    2014

  27. [35]

    and Cobbold, C

    Leinster, T. and Cobbold, C. A. (2012). Measuring diversity: the importance of species similarity. Ecology , 93(3):477--489

    2012

  28. [36]

    Lindley, D. V. (1956). On a Measure of the Information Provided by an Experiment . The Annals of Mathematical Statistics , 27(4):986--1005

    1956

  29. [37]

    Liu, J., Li, Z., Yao, Y., Xu, F., Ma, X., Xu, M., and Tong, H. (2022). Fair Representation Learning: An Alternative to Mutual Information . In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1088--1097

    2022

  30. [38]

    B., and G \'o mez-Gualdr \'o n, D

    Liu, T.-W., Nguyen, Q., Dieng, A. B., and G \'o mez-Gualdr \'o n, D. A. (2024). Diversity-driven, efficient exploration of a mof design space to optimize mof properties. Chemical Science , 15(45):18903--18919

    2024

  31. [39]

    MacKay, D. (1992a). Information-Based Objective Functions for Active Data Selection . Neural Computation

    1992

  32. [40]

    MacKay, D. (1992b). The Evidence Framework Applied to Classification Networks . Neural Computation

    1992

  33. [41]

    H., Murray, I., and Latham, P

    Macke, J. H., Murray, I., and Latham, P. (2011). How biased are maximum entropy models? In Advances in Neural Information Processing Systems , volume 24

    2011

  34. [42]

    Maes, F., Collignon, A., Vandermeulen, D., Marchal, G., and Suetens, P. (2002). Multimodality image registration by maximization of mutual information. IEEE transactions on Medical Imaging , 16(2):187--198

    2002

  35. [43]

    Marrelec, G., Mess \'e , A., and Bellec, P. (2015). A Bayesian Alternative to Mutual Information for the Hierarchical Clustering of Dependent Random Variables . PloS one , 10(9):e0137278

    2015

  36. [44]

    Meister, C., Giulianelli, M., and Pimentel, T. (2024). Towards a Similarity-adjusted Surprisal Theory . arXiv preprint . arXiv:2410.17676

    2024 arXiv

  37. [45]

    and Goodhill, G

    M \"o lter, J. and Goodhill, G. J. (2020). Limitations to Estimating Mutual Information in Large Neural Populations . Entropy , 22(4):490

    2020

  38. [46]

    and Khalili, N

    Mousavi, M. and Khalili, N. (2024). VSI: An Interpretable Bayesian Feature Ranking Method Based on Vendi Score . SSRN

    2024

  39. [47]

    I., Cavagnaro, D

    Myung, J. I., Cavagnaro, D. R., and Pitt, M. A. (2013). A Tutorial on Adaptive Design Optimization . Journal of Mathematical Psychology , 57(3-4):53--67

    2013

  40. [48]

    and Dieng, A

    Nguyen, Q. and Dieng, A. B. (2024). Quality-Weighted Vendi Scores And Their Application To Diverse Experimental Design . In International Conference on Machine Learning

    2024

  41. [49]

    Novick, A., Cai, D., Nguyen, Q., Garnett, R., Adams, R., and Toberer, E. (2024). Probabilistic Prediction of Material Stability: Integrating Convex Hulls into Active Learning . Materials Horizons

    2024

  42. [50]

    Oliver, D. S. (2022). Information content in 4d seismic data: Effect of correlated noise. Journal of Petroleum Science and Engineering , 208:109728

    2022

  43. [51]

    Paninski, L. (2003). Estimation of Entropy and Mutual Information . Neural Computation , 15(6):1191--1253

    2003

  44. [52]

    R., Treves, A., and Rolls, E

    Panzeri, S., Schultz, S. R., Treves, A., and Rolls, E. T. (1999). Correlations and the encoding of information in the nervous system. Proceedings of the Royal Society of London. Series B: Biological Sciences , 266(1423):1001--1012

    1999

  45. [53]

    P., Bencomo, G

    Pasarkar, A. P., Bencomo, G. M., Olsson, S., and Dieng, A. B. (2023). Vendi Sampling For Molecular Simulations: Diversity As A Force For Faster Convergence And Better Exploration . The Journal of Chemical Physics , 159(14)

    2023

  46. [54]

    Pasarkar, A. P. and Dieng, A. B. (2024). Cousins Of The Vendi Score: A Family Of Similarity-Based Diversity Metrics For Science And Machine Learning . In International Conference on Artificial Intelligence and Statistics , pages 3808--3816. PMLR

    2024

  47. [55]

    Pasarkar, A. P. and Dieng, A. B. (2025). The Vendiscope: An Algorithmic Microscope For Data Collections . arXiv preprint . arXiv:2502.10828

    2025 arXiv

  48. [56]

    Peng, H., Long, F., and Ding, C. (2005). Feature Selection Based on Mutual Information: Criteria of Max-Dependency, Max-Relevance, and Min-Redundancy . IEEE Transactions on Pattern Analysis and Machine Intelligence , 27(8):1226--1238

    2005

  49. [57]

    Peters, J., Janzing, D., and Sch \"o lkopf, B. (2017). Elements of causal inference: foundations and learning algorithms . The MIT Press

    2017

  50. [58]

    Petty, G. W. (2018). On Some Shortcomings of Shannon Entropy as a Measure of Information Content in Indirect Measurements of Continuous Variables . Journal of Atmospheric and Oceanic Technology , 35(5):1011--1021

    2018

  51. [59]

    and Cho, K

    Peyrard, M. and Cho, K. (2025). Meta-Statistical Learning: Supervised Learning of Statistical Inference . arXiv preprint . arXiv:2502.12088

    2025

  52. [60]

    and Garner, W

    Podgorny, P. and Garner, W. R. (1979). Reaction time as a measure of inter-and intraobject visual similarity: Letters of the alphabet. Perception & Psychophysics , 26(1):37--52

    1979

  53. [61]

    Pukelsheim, F. (2006). Optimal Design of Experiments . SIAM

    2006

  54. [62]

    and Panzeri, S

    Quian Quiroga, R. and Panzeri, S. (2009). Extracting information from neuronal populations: information theory and decoding approaches. Nature Reviews Neuroscience , 10(3):173--185

    2009

  55. [63]

    Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning . The MIT Press

    2006

  56. [64]

    Rezaei, M. R. and Dieng, A. B. (2025a). The alpha -Alternator: Dynamic Adaptation To Varying Noise Levels In Sequences Using The Vendi Score For Improved Robustness and Performance . arXiv preprint . arXiv:2502.04593

    2025 arXiv

  57. [65]

    Rezaei, M. R. and Dieng, A. B. (2025b). Vendi-RAG: Adaptively Trading-Off Diversity And Quality Significantly Improves Retrieval Augmented Generation With LLMs . arXiv preprint . arXiv:2502.11228

    2025 arXiv

  58. [66]

    Rhee, A., Cheong, R., and Levchenko, A. (2012). The application of information theory to biochemical signaling systems. Physical biology , 9(4):045011

    2012

  59. [67]

    Schroeder, M. J. (2004). An Alternative to Entropy in the Measurement of Information . Entropy , 6(5):388--412

    2004

  60. [68]

    K., Ahmad, A., Andreetto, M., Prabhakaran, V., Prabhu, U., Dieng, A

    Senthilkumar, N. K., Ahmad, A., Andreetto, M., Prabhakaran, V., Prabhu, U., Dieng, A. B., Bhattacharyya, P., and Dave, S. (2024). Beyond Aesthetics: Cultural Competence in Text-to-Image Models . Advances in Neural Information Processing Systems , 37:13716--13747

    2024

  61. [69]

    Settles, B. (2009). Active learning. Technical report, University of Wisconsin-Madison Department of Computer Sciences

    2009

  62. [70]

    Shannon, C. (1948). A Mathematical Theory of Communication . The Bell System Technical Journal , 27(3):379--423

    1948

  63. [71]

    and Hu, Y.-H

    Sheng, X. and Hu, Y.-H. (2004). Maximum Likelihood Multiple-Source Localization Using Acoustic Energy Measurements with Wireless Sensor Networks . IEEE transactions on signal processing , 53(1):44--53

    2004

  64. [72]

    N., Dastani, M., and van Ommen, T

    Simoes, F. N., Dastani, M., and van Ommen, T. (2024). Fundamental Properties of Causal Entropy and Information Gain . In Causal Learning and Reasoning , pages 188--208. PMLR

    2024

  65. [73]

    Slamecka, N. J. (1963). Choice Reaction-Time as a Function of Meaningful Similarity . The American Journal of Psychology , 76(2):274--280

    1963

  66. [74]

    B., Kirsch, A., Farquhar, S., Gal, Y., Foster, A., and Rainforth, T

    Smith, F. B., Kirsch, A., Farquhar, S., Gal, Y., Foster, A., and Rainforth, T. (2023). Prediction-Oriented Bayesian Active Learning . In International Conference on Artificial Intelligence and Statistics , pages 7331--7348. PMLR

    2023

  67. [75]

    and Ermon, S

    Song, J. and Ermon, S. (2020). Understanding the Limitations of Variational Mutual Information Estimators . In International Conference on Learning Representations

    2020

  68. [76]

    Torkkola, K. (2003). Feature Extraction by Non-Parametric Mutual Information Maximization . Journal of Machine Learning Research , 3:1415--1438

    2003

  69. [77]

    X., Epps, J., and Bailey, J

    Vinh, N. X., Epps, J., and Bailey, J. (2009). Information Theoretic Measures for Clusterings Comparison: Is a Correction for Chance Necessary? In International Conference on Machine Learning , pages 1073--1080

    2009

  70. [78]

    and Seeger, M

    Williams, C. and Seeger, M. (2000). Using the Nystr \"o m Method to Speed Up Kernel Machines . Advances in Neural Information Processing Systems 13

    2000

  71. [79]

    T., Borovitskiy, V., Terenin, A., Mostowsky, P., and Deisenroth, M

    Wilson, J. T., Borovitskiy, V., Terenin, A., Mostowsky, P., and Deisenroth, M. P. (2021). Pathwise Conditioning of Gaussian Processes . Journal of Machine Learning Research

    2021

  72. [80]

    Wu, S., Lu, K., Xu, B., Lin, J., Su, Q., and Zhou, C. (2023). Self-Evolved Diverse Data Sampling for Efficient Instruction Tuning . arXiv preprint . arXiv:2311.08182 [cs.CL]

    2023 arXiv

  73. [81]

    Xu, Y., Zhao, S., Song, J., Stewart, R., and Ermon, S. (2020). A Theory of Usable Information under Computational Constraints . In International Conference on Learning Representations

    2020

  74. [82]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  75. [83]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  76. [84]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.