{"id":"860f160e-faab-468b-bffd-c5c1601b6aa8","arxiv_id":"2505.09007","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"VIG, defined as the difference between marginal and conditional Vendi entropy, is proposed as a sample-based, similarity-aware alternative to mutual information, with applications in active learning and level-set estimation.","lead":"This paper introduces Vendi Information Gain (VIG), a new measure of information gain that replaces Shannon entropy with the Vendi Score, which accounts for similarity between samples and works directly from samples without a tractable density. It applies VIG to model human response times, active data acquisition, and level-set estimation, claiming advantages over mutual information in high-dimensional settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's 'Boundedness' claim is false: for q>1, VIG can be negative even under the identity kernel where it reduces to a Rényi entropy difference, so the central 'valid information gain' property does not hold as stated.","rationale":"The reader's conditional verdict is based on two concerns: sensitivity to the kernel and unproven nonnegativity. I agree with the nonnegativity concern and sharpen it into an exact counterexample: for q>1, with the identity kernel and probability-weighted Vendi entropy, VIG can be negative. This is directly relevant to the central claim that VIG is a valid information gain measure, because the paper states boundedness as a general property without restricting q. It is also not a merely finite-sample artifact: the counterexample uses exact distributions. However, the main experimental claims in the paper are made with q=1, and the q=1 case may still satisfy nonnegativity under suitable conditions, so the result does not overturn the paper entirely. The appropriate verdict remains CONDITIONAL: the authors should restrict the boundedness claim, prove it for the regime they defend, and clarify the status of the q parameter. I do not see a reason to move to REJECT because the core construction and the q=1 empirical results are not refuted by this counterexample, and the issue is addressable with a qualification or a proof. The reader did not present the explicit q>1 counterexample, so my agreement is partial rather than full.","tokens_in":20420,"tokens_out":23949,"duration_ms":257103,"concrete_test":"Implement VIG as in Eqs. (2)-(4) with K=I_3, q=2, and the exact discrete distributions above: θ∈{1,2,3}, Y∈{0,1}, P(Y=0)=P(Y=1)=1/2, P(θ|Y=0)=(0.936,0.03,0.034), P(θ|Y=1)=(0.864,0.07,0.066). Compute VIG = H_2(marginal) - 0.5*H_2(cond0) - 0.5*H_2(cond1). A negative result (≈-7e-4) confirms the Section 3.2 'Boundedness' assertion is false. For a finite-sample robustness check, draw 10^5 i.i.d. pairs from this joint distribution, compute empirical frequency matrices with the identity kernel, and verify that the sample VIG for q=2 is also negative for sufficiently large N.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2, 'Boundedness', asserts without proof that VIG ranges between 0 and the initial entropy, 'similar to MI'. This is a load-bearing property: a quantity that can be negative is not a valid information gain in the usual sense. The assertion is not merely unproven; it is false for q>1. Take a discrete θ with three outcomes and Y binary with P(Y=0)=P(Y=1)=1/2, P(θ|Y=0)=(0.936,0.03,0.034), P(θ|Y=1)=(0.864,0.07,0.066). The marginal is P(θ)=(0.9,0.05,0.05). Use the identity kernel K=I_3 and the probability-weighted Vendi entropy of Eqs. (2)-(3) with q=2, exactly the setting the paper uses to show VIG reduces to MI (except with q=1). Then H_2(θ)=-ln(0.815)≈0.204, while E_Y[H_2(θ|Y)]=0.5[-ln(0.878152)-ln(0.755752)]≈0.205. Thus VIG≈-7×10^{-4}<0. Since q is a tunable hyperparameter throughout the paper, including the ablation studies in Appendix B, the unqualified boundedness claim and the associated claim that VIG is a valid information gain measure are false as stated. This does not refute the q=1 experiments, but it shows the central property needs a qualification (q=1, or additional kernel/order conditions) and a proof.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Vendi Information Gain (VIG), defined in Eq. (4) as the expected reduction in Vendi entropy, VIG(θ,y;q) = H_V(D;q) − E_y[H_V(D_y;q)], where H_V is the Rényi entropy of the normalized eigenvalues of a (possibly probability-weighted) similarity kernel matrix. The authors claim that VIG generalizes mutual information, recovers MI for the identity kernel at q=1, is bounded between 0 and the initial entropy, is asymmetric, and is additive under independence. They illustrate these properties on communication channels and in three application areas: modeling human response times, active data acquisition (step function learning, death processes, location sensing), and level-set estimation with Gaussian-process surrogates. Main experiments use q=1, while Appendix B reports ablations over q.","tokens_in":20768,"tokens_out":8251,"duration_ms":81834,"significance":"If the theoretical claims are corrected, VIG is a useful sample-based, similarity-aware alternative to mutual information. The derivation that VIG equals MI for the identity kernel at q=1 is clean and parameter-free, and the level-set estimation experiments show promising practical potential. The paper does not ship code at review time, and the response-time simulation sets similarity values by hand, so the empirical component is a demonstration rather than a full validation. The main weaknesses are mathematical: the boundedness claim is false as stated for q>1, and the additivity proof is incomplete. These issues are load-bearing for the characterization of VIG as a valid information-gain measure and need to be fixed before the central claims are accepted.","major_comments":[{"comment":"The claim that VIG ranges between 0 and the initial entropy is asserted without proof and is false for q>1, even under the identity kernel where the probability-weighted Vendi entropy of Eqs. (2)-(3) reduces to the Rényi entropy of the distribution. Concretely, take θ with three outcomes and Y binary with P(Y=0)=P(Y=1)=1/2, P(θ|Y=0)=(0.936,0.03,0.034), P(θ|Y=1)=(0.864,0.07,0.066), and marginal P(θ)=(0.9,0.05,0.05). With K=I and q=2, H_2(θ)=−ln(0.815)≈0.204, while E_Y[H_2(θ|Y)] = 0.5[−ln(0.878152)−ln(0.755752)]≈0.205, so VIG≈−7×10^{-4}<0. Since q is a free hyperparameter, including the ablation values q=0.1,0.2,2,∞ in Appendix B, the unqualified boundedness claim, and the associated statement that VIG is a valid information-gain measure, are false as stated. The authors should restrict the claim to q=1, prove sufficient conditions on q and the kernel for nonnegativity, or qualify the claim explicitly.","section":"Section 3.2, Boundedness"},{"comment":"Equation (8) states that under conditional independence VIG((x,y),z) = VIG(x,z) + VIG(y,z), but the proof in Eqs. (9)-(12) only establishes H_V(D_{x,y};q) = H_V(D_x;q) + H_V(D_y;q) for unconditional independence. The required conditional statement H_V(D_{x,y|z};q) = H_V(D_{x|z};q) + H_V(D_{y|z};q) is never shown. In addition, the decomposition K_{x,y}=K_x⊗K_y in Eq. (10) is an assumption about the joint kernel, not a consequence of independence; it must be stated explicitly as a product-kernel assumption. As written, Eq. (8) is unproved.","section":"Section 3.2, Additivity under independence, Eqs. (8)-(12)"},{"comment":"The paper claims that VIG estimates are stable where MI estimates fail, but it provides no finite-sample statistical characterization of the plug-in estimator \\widehat{VIG} = \\widehat{H}_V(D) − E_y[\\widehat{H}_V(D_y)]. No bias, variance, consistency, or sample-complexity result is stated, and no conditions are given on the kernel bandwidth or the number of samples. Given that the paper explicitly motivates VIG as a sample-based quantity, a theoretical or at least a systematic empirical characterization of the estimator is needed to support the claim that VIG 'does not degenerate to 0 with decreasing samples like MI'.","section":"Section 4 and Figure 1"},{"comment":"The paper acknowledges in Section 6 that kernel choice depends on the user, but none of the experiments reports a sensitivity analysis over the Gaussian kernel bandwidth or compares alternative kernels. Because VIG's values and rankings depend directly on the kernel, the claimed advantages over MI in the applications (Figures 2, 4, 5) may not be robust to this choice. A sensitivity analysis over kernel hyperparameters is needed to substantiate the general claim that VIG is a reliable information-gain measure.","section":"Section 6 and Section 4"}],"minor_comments":[{"comment":"The text says 'the identify kernel'; this should be 'the identity kernel'.","section":"Section A.2"},{"comment":"The statement that VIG equals 0 when θ does not depend on y holds only at the population level or in expectation; for finite samples, the empirical VIG will generally not be exactly zero even under independence. The text should state this qualification.","section":"Section 3.2, Reducing to 0 with independent variables"},{"comment":"The response-time simulation sets the similarity parameter s by hand (s=0, 0.5, 0.9) and reports agreement with Slamecka (1963); because s is not estimated from data, the match is a demonstration rather than a validation. This should be stated explicitly in the main text, not only in Appendix A.","section":"Section 4.1 and Eq. (23)"},{"comment":"There is a typographical error in the definition of the probability-weighted kernel matrix: 'diag(pp)' should be 'diag(p)'.","section":"Eq. (2)"},{"comment":"The appendix says each run is repeated 10 times, while the caption of Figure 4 reports averages across 50 repeats. Please clarify which number is correct.","section":"Section A.4 and Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The main technical obstacle is the false boundedness claim for q>1; this is fixable within the scope of the paper by restricting the claim to q=1 or adding explicit conditions and proofs. The additivity gap is also fixable with a product-kernel assumption and a proof of the conditional decomposition. I do not see grounds for rejection: the q=1 experiments are not refuted by the q>1 counterexample, but the mathematical framing needs substantial revision. Please also ask for kernel-sensitivity experiments and a clarification of the finite-sample claims. The literature review is broad, and the self-citations to prior Vendi Score work are appropriate given the direct lineage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper defines VIG, a sample-based information gain that generalizes mutual information via the Vendi score's entropy, and applies it to active learning, response-time modeling, and level-set estimation. The core definition and the MI-recovery proof under identity kernels at q=1 are correct, and the communication-channel example is a compact, convincing illustration of how similarity changes information gain.\n\nWhat is actually new: VIG as a named quantity, and the VIG-based active acquisition and level-set estimation framework. The experiments in Section 4 cover several realistic tasks and show VIG performing competitively or better than MI, especially in LSE. The response-time simulation is a nice bridge to cognitive science, even though the similarity values are hand-set.\n\nThe soft spots are real, and one is load-bearing. Section 3.2's 'Boundedness' claim is false as stated for q>1. I checked the stress-test counterexample: with the identity kernel and a three-outcome variable, VIG with q=2 is about -7e-4. The unqualified statement that VIG ranges between 0 and the initial entropy fails. Since the paper ablates q beyond 1, the claim needs either a proof under restricted conditions or explicit qualification to q=1. This does not invalidate the main q=1 experiments, but it means the theoretical claim as written is wrong. Relatedly, the discussion that Vendi entropy is strictly positive and so VIG does not have the negative-information problem is misleading, because VIG itself can be negative.\n\nOther issues: the additivity argument proves the decomposition for the marginal Vendi entropy, not for the conditional term, so the additivity of VIG itself is not fully established. No code or simulated data is released, which makes the empirical results hard to reproduce. The kernel choice is acknowledged to be user-dependent, but there is no sensitivity analysis over bandwidth or other kernel hyperparameters. MI baseline details are also underspecified. These are all fixable. The 'first-ever' phrasing in Section 4.2 is overclaiming but minor.\n\nThe citation pattern is fine; the paper engages with prior similarity-aware alternatives rather than ignoring them.\n\nWho this is for: people working on sample-based information measures, active learning, or level-set estimation. It deserves a serious referee. Recommendation: send to peer review, with a request to fix the boundedness claim, release code, add kernel sensitivity analysis, and tighten the novelty claims.","headline":"A useful sample-based information measure with a correct q=1 MI-recovery proof, but the unqualified boundedness claim is false for q>1 and the paper needs code and kernel sensitivity analysis.","tokens_in":21296,"tokens_out":3610,"would_cite":true,"duration_ms":34735,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["94A17"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes Vendi Information Gain, a sample-based, similarity-aware information gain measure that reduces to mutual information when samples are completely dissimilar, and shows it remains stable where MI fails.","keywords":["Vendi Information Gain","mutual information","Vendi Score","information gain","similarity kernel","active data acquisition","level-set estimation","Rényi entropy"],"falsifier":"Compute VIG for a fixed two-class data set under a Gaussian kernel with two very different bandwidths, one much smaller than the typical inter-sample distance and one comparable to the class separation; if the ordering of candidate queries or channels reverses, the claim that VIG reliably captures information gain fails. Equivalently, a single finite-sample example where independent variables give $\\mathrm{VIG} < 0$ would contradict the paper's boundedness claim.","tokens_in":20189,"feed_emoji":"ℹ️","tokens_out":7636,"duration_ms":74449,"temperature":0.7,"pith_summary":"The paper's central claim is that information gain can be quantified by Vendi Information Gain (VIG): the expected drop in Vendi entropy—the Rényi entropy of the eigenvalues of a similarity kernel matrix—of a variable after observing another. Unlike Shannon's mutual information, VIG is computed directly from samples, requires no tractable density, is asymmetric, and incorporates pairwise similarity between sample values. VIG reduces exactly to mutual information when all samples are treated as completely dissimilar and the order is $q=1$, so MI is a special case. If correct, VIG would provide an information-theoretic criterion that remains usable where MI is intractable or misleading, including active data acquisition and level-set estimation.","feed_headline":"Vendi Information Gain makes information measurable without densities","feed_subtitle":"A sample-only, similarity-aware measure that reduces to mutual information when samples are distinct.","key_machinery":"The key object is the Vendi score and its logarithm, the Vendi entropy, defined as the Rényi entropy of the normalized eigenvalues of a positive semidefinite similarity kernel matrix $K$ built from samples. VIG is the expected reduction in this Vendi entropy after conditioning on another variable. The kernel is what carries the argument: it encodes which samples count as similar, so the resulting information gain respects distances rather than treating every distinct outcome as equally separated. The identity kernel collapses Vendi entropy to ordinary Rényi entropy of the sample probabilities, which is exactly the step that makes VIG recover mutual information at $q=1$.","core_discovery":"The paper defines Vendi information gain as $\\mathrm{VIG}(\\theta, y; q) = H_V(\\mathcal{D}; q) - \\mathbb{E}_y[H_V(\\mathcal{D}_y; q)]$, where $H_V$ is Vendi entropy, the Rényi entropy of normalized eigenvalues of a kernel matrix on samples. The central discovery is that this quantity is a valid, directional information-gain measure that accounts for similarity: when the kernel matrix is the identity, Vendi entropy becomes Rényi entropy, and at order $q=1$ VIG equals $I(\\theta; y)$. The paper further claims that VIG is zero for independent variables, is bounded between zero and the initial Vendi entropy, is additive under independent variables, and is sensitive to the spread of the variable's distribution; it also demonstrates that VIG estimates remain stable where MI estimates become biased or degenerate.","pith_inferences":["A testable extension the paper leaves implicit: because VIG is asymmetric, comparing $\\mathrm{VIG}(\\theta; y)$ with $\\mathrm{VIG}(y; \\theta)$ could serve as a directional dependence diagnostic in causal or directed-information settings, though the paper only gestures at this.","The paper reports no sensitivity analysis over kernel bandwidth; a natural check is whether the channel-ordering and active-learning rankings shown in its figures survive small perturbations of the Gaussian lengthscale. If rankings flip, the practical advantage over MI is kernel-tuned rather than structural.","Because VIG is computed entirely from posterior samples, it could be plugged into any sampling-based acquisition scheme, including those using neural-network posteriors or implicit level-set thresholds; the paper mentions such connections but does not test them.","A stronger validation would be on synthetic problems where the true information gain is known in closed form: fixing a kernel, one could compare VIG's ranking of candidate queries against the exact MI ranking to see whether similarity-sensitivity always improves the measure or only helps in multimodal regimes."],"forward_implications":["Any pipeline that currently approximates MI from samples in high dimensions can instead use VIG without density estimation, and the paper's experiments show VIG estimates do not collapse to zero as sample size shrinks.","Because VIG is asymmetric, it can model directional information flow in communication, causal, and cognitive settings where MI's symmetry is a known limitation.","VIG's additivity under independence lets a joint system's Vendi entropy decompose into a sum of individual entropies, so computing VIG can be reduced to smaller eigendecompositions.","VIG yields a concrete, sample-only active data acquisition and level-set estimation policy; in the paper's experiments it consistently achieves higher F1 scores than MI, STRADDLE, LSE, and uncertainty sampling.","Setting $q=1$ and the identity kernel recovers MI, so VIG is a strict generalization rather than a competing ad hoc quantity."],"supporting_citations":[{"why":"Defines mutual information as 'rate of transmission' and supplies the baseline measure that VIG replaces.","marker":"Shannon, 1948"},{"why":"Provides the standard textbook treatment of mutual information whose entropy-difference definition frames the comparison.","marker":"Cover, 1999"},{"why":"Introduces the Vendi Score, the similarity-based diversity metric whose logarithm VIG uses as entropy.","marker":"Friedman and Dieng, 2023"},{"why":"Extends the Vendi Score family and explains how the order $q$ controls sensitivity to rarity, justifying the role of $q$ in VIG.","marker":"Pasarkar and Dieng, 2024"},{"why":"Provides the human response-time neural decoding model that VIG plugs into as the stopping criterion.","marker":"Christie et al., 2023"},{"why":"Supplies the disease-hotspot data sets used in the level-set estimation experiments.","marker":"Andrade-Pacheco et al., 2020"},{"why":"Defines the STRADDLE baseline policy that VIG is compared against in level-set estimation.","marker":"Bryan et al., 2005"},{"why":"Defines the 'LSE' baseline policy that VIG is compared against in level-set estimation.","marker":"Gotovos et al., 2013"}],"fun_headline_variants":["Vendi Information Gain: sample-only, similarity-aware info measure","A similarity-based information gain that needs only samples","VIG: mutual information without densities, with similarity","No densities, no symmetry: Vendi Information Gain for science"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire measure depends on a user-chosen similarity kernel and its hyperparameters; the paper acknowledges this choice is fully up to the user, and if rankings of information gain change under reasonable kernel choices, VIG's advantage over MI is not intrinsic.","fun_headline_variants_meta":{"raw":{"variants":["Vendi Information Gain: sample-only, similarity-aware info measure","A similarity-based information gain that needs only samples","VIG: mutual information without densities, with similarity","No densities, no symmetry: Vendi Information Gain for science"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001027,"raw_usage":{"total_tokens":4379,"prompt_tokens":1044,"completion_tokens":3335,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":3269}},"tokens_in":660,"tokens_out":3335,"duration_ms":22119,"temperature":1.0,"reasoning_tokens":3269,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:42:47.083353+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute VIG for a fixed two-class data set under a Gaussian kernel with two very different bandwidths, one much smaller than the typical inter-sample distance and one comparable to the class separation; if the ordering of candidate queries or channels reverses, the claim that VIG reliably captures information gain fails. Equivalently, a single finite-sample example where independent variables give $\\mathrm{VIG} < 0$ would contradict the paper's boundedness claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines mutual information as 'rate of transmission' and supplies the baseline measure that VIG replaces."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the standard textbook treatment of mutual information whose entropy-difference definition frames the comparison."},{"cited_title":"T., Johnson, H","cited_arxiv_id":null,"evidence_quote":"Provides the human response-time neural decoding model that VIG plugs into as the stopping criterion."},{"cited_title":"C., Genovese, C","cited_arxiv_id":null,"evidence_quote":"Defines the STRADDLE baseline policy that VIG is compared against in level-set estimation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the 'LSE' baseline policy that VIG is compared against in level-set estimation."}],"review_version":1}