{"id":"63a144cd-c1e5-4334-8cab-01d3ac7e86ee","arxiv_id":"2507.15372","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Cross mutual information is a new measure that averages the pointwise mutual information of a reference distribution over test samples from another distribution.","lead":"The paper proposes 'cross mutual information', a measure that scores how strongly the dependence between two variables in a reference dataset appears in a new dataset. It targets non-stationary or online data, such as brain signals across different task states.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The online significance test in Appendix A has a degenerate null: shuffling reference x makes every surrogate CI zero, so the paper's inferential claim for online data is unsupported.","rationale":"I read the paper in good faith and agree that Eq. (4) defines a mathematically sound measure with useful properties, including the Gaussian form in Appendix B. The reader's weakest assumption about reference support is legitimate and the paper itself acknowledges it. However, the more load-bearing flaw is the significance-testing procedure for online data. The cross MI is only nonzero when the reference distribution carries X-Y dependence, so shuffling reference x in the limited-test-data case destroys the quantity being measured and yields a degenerate null distribution of exactly zero. This makes the online significance claim internally inconsistent, not merely dependent on an external assumption. The concrete test would demonstrate the uncontrolled false-positive rate directly. Since the reader already returned CONDITIONAL for reasons including the significance-testing issue, my analysis does not change the overall verdict; it sharpens the justification for that verdict.","tokens_in":11446,"tokens_out":3905,"duration_ms":46786,"concrete_test":"Simulate reference data with a strong linear X-Y dependence and test data in which X and Y are independent but inside the reference support. Estimate CI with KSG and apply the Appendix A reference-shuffle significance test using 200 surrogates, repeated over 1000 test datasets. A valid test should reject at approximately the nominal rate; the degenerate test will reject essentially always because every surrogate CI is exactly 0. For comparison, repeat with large test samples and shuffle the test x, which should produce the correct false-positive rate. This settles whether the online significance procedure is usable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim includes using cross MI for online data where conventional MI cannot be reliably estimated. But the significance test proposed for exactly that regime is degenerate and tests the wrong null. In Appendix A, when test data are limited, the paper block-shuffles the reference x instead of the test x. Because CI_pq in Eq. (4) is nonzero only when the reference distribution q encodes X-Y dependence, replacing q(x,y) with q(x)q(y) makes every surrogate local MI identically zero: log(q(x)q(y)/(q(x)q(y))) = 0. The resulting null distribution is a point mass at zero. Any nonzero estimated CI, including pure estimation noise, is then declared statistically significant, so false positives are not controlled. This is an internal inconsistency in a load-bearing application: the measure itself is well-defined, but the paper's inferential procedure for online data cannot support its claimed use. The reference-support limitation flagged by the reader is real but is explicitly acknowledged in Sec. IV-A and Fig. 3A; the degenerate null is not acknowledged and is more damaging because it invalidates the proposed test rather than merely limiting applicability.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces cross mutual information (cross MI), defined in Eq. (4) as CI_pq = E_{(x,y)~p(x,y)}[log(q(x,y)/(q(x)q(y)))], which measures how strongly the X-Y dependence encoded in a reference distribution q is expressed in test data sampled from p. The authors derive basic properties (CI_pp = I_p, zero when q factorizes, possible negativity), present simulation studies with KSG estimation for supported and unsupported reference distributions, give a Gaussian closed form, and relate the measure to linear-regression residuals. They also propose permutation significance tests for MI, differences in MI, and cross MI, with the stated goal of handling non-stationary and online data. The mathematical definition and the Gaussian derivation are straightforward and appear correct.","tokens_in":11596,"tokens_out":7927,"duration_ms":100105,"significance":"If taken as a descriptive statistic, cross MI is a useful, parameter-free way to compare pairwise dependence across conditions without estimating the test distribution, and the paper provides an analytic Gaussian form plus public code. The paper's stronger inferential claim, however, depends on the significance test in Appendix A, and that test is degenerate for the limited-test-data (online) setting that motivates the paper. The core measure may still be valuable, but the statistical procedure needs repair before the online-data claims can be supported.","major_comments":[{"comment":"The permutation test proposed for limited test data is degenerate and tests the wrong null. In Eq. (4), CI_pq is an expectation over p of log(q(x,y)/(q(x)q(y))). If the reference x is block-shuffled, the surrogate reference joint is, up to finite-sample estimation error, q(x)q(y), so every surrogate pointwise term is log(q(x)q(y)/(q(x)q(y))) = 0. The null distribution therefore collapses to a point mass at zero (or, with a plug-in estimator, to estimator noise around zero). Any non-zero estimate of CI_pq, including pure estimator noise, will be declared significant, so false positives are not controlled. This invalidates the online-data significance testing advertised in Sec. I and used in Fig. 1C ('not statistically significant (see Appendix A)'). A valid test of 'no dependence in the test data' must permute the test x-y pairing while keeping the reference q fixed; when only one test point is available, no nonparametric null can be formed in this way, and the text should either provide an alternative procedure or explicitly withdraw the inferential claim for single online points.","section":"Appendix A, 'Cross MI: is |CI_pq| non-zero?'"}],"minor_comments":[{"comment":"The word 'Cruicially' should be 'Crucially'.","section":"Sec. II-B"},{"comment":"'Difference choices for the reference data' should be 'Different choices for the reference data'.","section":"Fig. 2 caption"},{"comment":"The phrase 'the the X-Y dependence' contains a duplicated 'the'.","section":"Sec. IV-C"},{"comment":"The proposed nearest-neighbour distance check for whether the reference distribution supports the test data is informal; specifying a concrete threshold or validation procedure would make the limitation operational.","section":"Sec. IV-A"},{"comment":"The notation alternates between CI_pq and Cpq; using a single symbol consistently throughout would improve readability.","section":"Sec. III and Fig. 1"},{"comment":"The sign-flip procedure randomly assigns system labels with probabilities proportional to block counts; this is not the usual exchangeability permutation and should be justified or replaced by a standard permutation of pooled blocks.","section":"Appendix A, 'Differences in MI'"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The definitional core of the paper is sound and the simulations illustrate the measure's behavior, but the limited-test-data significance test in Appendix A is degenerate and undermines the paper's central online-data application. This is fixable by changing the permutation procedure or by explicitly restricting significance claims to settings with enough test samples. I would encourage the authors to address this before publication; the paper is otherwise within scope for cs.IT and the broader computational neuroscience audience."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The cross MI itself is a simple, natural, and genuinely new quantity: an expectation of pointwise MI under a different distribution, analogous to cross entropy. The paper does a good job of distinguishing it from prior 'cross MI' definitions, and the Gaussian form in Eq. (10) follows correctly by substitution. The derivation of the link to linear regression is clear, and the code is available. That part is solid.\n\nThe load-bearing problem is the statistical significance test in Appendix A for online data. When test data are limited, the paper shuffles the reference x rather than the test x. That constructs a null under which the reference distribution has no X-Y dependence, not a null under which the test data express no dependence relative to the reference. If q has strong dependence, CI_pq will be large for essentially any test point, even one drawn from an independent p. The null distribution may not be exactly a point mass — finite-sample estimation will add some spread — but the test is answering the wrong question. Since the online application is one of the paper's headline claims, this is a serious flaw. The paper should either develop a test that preserves the reference dependence while breaking the test dependence (when that is possible), or explicitly state that the procedure only checks whether the reference contains dependence, and adjust the interpretation accordingly.\n\nThe uniqueness claim in Appendix B — that cross MI is the unique measure of prediction quality satisfying certain axioms — is asserted rather than proven. It reads as a plausible conjecture, but the paper should either provide a proof or label it as one. This is a moderate issue, not fatal.\n\nTwo smaller points. The simulations generally lack error bars; Fig. 3A gives one standard deviation, but the other figures would benefit from confidence intervals or repeated-run comparisons. And the reference-support limitation in Sec. IV-A is real but explicitly acknowledged; the paper handles it honestly.\n\nThe measure itself is useful and likely to be cited by people comparing dependencies across conditions in neuroimaging or online settings. But the significance-testing flaw needs to be fixed before the online procedures can be trusted.\n\nRecommendation: send to peer review. A serious referee should push for a corrected significance test and a proof or caveated uniqueness claim, but the core definition and derivations deserve the attention.","headline":"Cross MI is a clean, well-defined new measure, but the Appendix A significance test for online data tests the wrong null and undercuts the paper's headline application.","tokens_in":12171,"tokens_out":2485,"would_cite":true,"duration_ms":32113,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["94A17","62B10"],"pacs":[],"model":"deepseek-v4-flash","headline":"Cross mutual information measures how a relationship in one dataset appears in another.","keywords":["cross mutual information","mutual information","information theory","non-stationary data","online data","conditional mutual information","linear regression","dependence measure"],"falsifier":"Simulate reference data with a linear dependence $y = 0.5x + \\text{noise}$ and test data with a sinusoidal dependence over a similar x-range: the paper's Fig. 3B predicts a strongly negative cross MI, showing that any X-Y dependence not present in the reference is read as surprise rather than dependence. A second check: push test data outside the reference's x-range (as in Fig. 3A) and estimate cross MI with a model-free estimator; the estimate should show high variance (the paper reports standard deviation 0.26), confirming the support assumption.","tokens_in":11223,"feed_emoji":"🔗","tokens_out":4922,"duration_ms":50066,"temperature":0.7,"pith_summary":"This paper proposes cross mutual information (cross MI), a measure of how strongly the X-Y dependence defined by one probability distribution q is expressed in new test data sampled from another distribution p. The quantity is $\\mathrm{CI}_{pq} = \\mathbb{E}_{p}[\\log(q(x,y)/(q(x)q(y)))]$, and unlike conventional mutual information it does not require estimating the test distribution, so it can be applied to a single online data point or to short non-stationary windows. The paper shows via simulations that cross MI equals MI when p=q, is zero when the reference has no dependence, can be negative, and that conditioning effects show up as synergy or redundancy relative to the reference. It also connects cross MI to linear regression, showing that it is a normalised, axiomatically motivated version of comparing squared residuals in cross-validation.","feed_headline":"Cross MI scores dependence through a reference distribution","feed_subtitle":"Works with a single online sample and separates real dependence from statistical surprise.","key_machinery":"The machinery is the pointwise (local) mutual information under the reference distribution, $i_q(x;y) = \\log(q(x,y)/q(x)q(y))$, averaged over test samples: $\\mathrm{CI}_{pq} = \\mathbb{E}_{p}[i_q]$. This expectation converts a reference model q into a scoring function for arbitrary test data; when q is Gaussian, $i_q$ reduces to a baseline term $-\\frac{1}{2}\\log(1-\\rho^2)$ plus a correction comparing the squared residuals of the posterior model (y given x) and the prior model (y alone), which is how the connection to linear regression arises.","core_discovery":"The central discovery is that the cross mutual information defined by $\\mathrm{CI}_{pq} = \\mathbb{E}_{x,y\\sim p}[\\log(q(x,y)/q(x)q(y))]$ gives the expected strength of dependence between X and Y in test samples drawn from p, as judged by a reference model q. Because only q needs to be well-estimated, the measure works when p is represented by few samples or a single point; the test data enters only through the empirical average of the pointwise (local) MI computed under q. The paper demonstrates in simulation that cross MI can separate the dependence expressed in a conditioned subset of a system from the dependence of the full ensemble, that it can go negative when test data is 'surprising' relative to q, and that a Gaussian model of q makes cross MI a direct generalisation of cross-validated sum-of-squared-residuals while retaining the chain rule.","pith_inferences":["One testable extension is change-point detection: with a fixed reference q, a running cross MI over short windows should rise or fall sharply when the system switches condition, and the sign pattern may identify which condition was entered.","Since cross MI is an expectation of a local score, it can be decomposed per sample; the distribution of those sample-level scores, not just the mean, may distinguish typical from atypical test data more sensitively than the scalar CI.","If cross MI is computed with a reference built from a long 'in the wild' recording, it could serve as a normalisation for task-based neuroimaging connectomics: each task condition is scored against the ensemble rather than analysed in isolation, and comparisons across subjects would require careful matching of reference supports.","The ratio sensitivity noted in Appendix C suggests a practical caution: when test data is folded into the reference, cross MI changes with the number of test samples, so comparing two systems requires equal-size or weighted references."],"forward_implications":["With a fixed reference distribution, cross MI can be tracked online, sample by sample, without waiting to accumulate enough test data to estimate p.","Comparing cross MI across conditions removes the synergy and redundancy distortions that conditional MI inherits from conditioning on the condition variable.","A negative cross MI signals that the test data is surprising relative to the reference, either because it lies outside the reference's support or follows a different dependency.","For Gaussian reference models, cross MI provides a normalised version of cross-validated prediction error that satisfies the chain rule over predictors.","The construction extends directly to conditional MI, transfer entropy, and active information storage, yielding 'cross' versions of those measures."],"supporting_citations":[{"why":"Supplies the standard definition of MI, its non-negativity lower-bound proof, and the Gaussian form $-\\frac{1}{2}\\log(1-\\rho^2)$ used to interpret cross MI.","marker":"[1]"},{"why":"The KSG estimator used for all MI and cross MI computations in the simulations.","marker":"[2]"},{"why":"Defines local (pointwise) MI whose expectation defines both MI and cross MI, including the axioms that make it the unique measure of prediction quality.","marker":"[3]"},{"why":"Provides the information-decomposition framework used to interpret conditioning effects as synergy (Fig. 2A/B) and redundancy (Fig. 2C).","marker":"[5]"},{"why":"Extends the decomposition perspective for target effects, supporting the claim that conditioning can add synergy or remove redundancy.","marker":"[6]"},{"why":"Motivates the block-permutation significance tests used for autocorrelated time series in the paper's simulation studies.","marker":"[12]"},{"why":"Software toolkit (JIDT) used to compute MI and cross MI, making the reported numbers reproducible.","marker":"[16]"},{"why":"Supports the statement that pointwise MI quantifies how a sample constrains the expected probabilities, backing the quality-of-prediction interpretation.","marker":"[17]"}],"fun_headline_variants":["Cross MI measures dependence via a reference set","Dependence in test samples, judged by a reference","Cross MI: compare dependence across samples","New metric for comparing dependence via a reference","Cross MI separates real dependence from surprise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a well-sampled reference distribution q is available that fully spans the accessible regions of the system's probability space, so that test data falls inside the reference's support; otherwise cross MI becomes sensitive to the estimator and unreliable.","fun_headline_variants_meta":{"raw":{"variants":["Cross MI measures dependence via a reference set","Dependence in test samples, judged by a reference","Cross MI: compare dependence across samples","New metric for comparing dependence via a reference","Cross MI separates real dependence from surprise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001032,"raw_usage":{"total_tokens":4306,"prompt_tokens":866,"completion_tokens":3440,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":3374}},"tokens_in":482,"tokens_out":3440,"duration_ms":24659,"temperature":1.0,"reasoning_tokens":3374,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:33:06.267169+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate reference data with a linear dependence $y = 0.5x + \\text{noise}$ and test data with a sinusoidal dependence over a similar x-range: the paper's Fig. 3B predicts a strongly negative cross MI, showing that any X-Y dependence not present in the reference is read as surprise rather than dependence. A second check: push test data outside the reference's x-range (as in Fig. 3A) and estimate cross MI with a model-free estimator; the estimate should show high variance (the paper reports standard deviation 0.26), confirming the support assumption.","supporting_citations":[{"cited_title":"Wiley, 2006","cited_arxiv_id":null,"evidence_quote":"Supplies the standard definition of MI, its non-negativity lower-bound proof, and the Gaussian form $-\\frac{1}{2}\\log(1-\\rho^2)$ used to interpret cross MI."},{"cited_title":"Esti- mating mutual information","cited_arxiv_id":null,"evidence_quote":"The KSG estimator used for all MI and cross MI computations in the simulations."},{"cited_title":"”Transmission of information: A statistical theory of communications.” American Journal of Physics 29.11 (1961): 793-794","cited_arxiv_id":null,"evidence_quote":"Defines local (pointwise) MI whose expectation defines both MI and cross MI, including the axioms that make it the unique measure of prediction quality."},{"cited_title":"”Information decomposition of target effects from multi-source interactions: Perspectives on previous, current and future work.” Entropy 20.4 (2018): 307","cited_arxiv_id":null,"evidence_quote":"Extends the decomposition perspective for target effects, supporting the claim that conditioning can add synergy or remove redundancy."},{"cited_title":"”Assessing the significance of directed and multivariate measures of linear dependence between time series.” Physical Review Research 3.1 (2021): 013145","cited_arxiv_id":null,"evidence_quote":"Motivates the block-permutation significance tests used for autocorrelated time series in the paper's simulation studies."},{"cited_title":"JIDT: An information-theoretic toolkit for studying the dynamics of complex systems","cited_arxiv_id":null,"evidence_quote":"Software toolkit (JIDT) used to compute MI and cross MI, making the reported numbers reproducible."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the statement that pointwise MI quantifies how a sample constrains the expected probabilities, backing the quality-of-prediction interpretation."}],"review_version":1}