{"id":"fdbf48a4-0a86-4ba6-a26c-2655bc5907f7","arxiv_id":"2412.03584","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"ResMI measures clustering similarity via the mutual information between pair-co-membership indicators, claiming constant baseline and model independence.","lead":"This paper introduces ResMI, a new score for comparing two clusterings, built from random pairs of items and mutual information. Generalist readers may care because scoring cluster quality is a common need, and ResMI is designed to avoid biases of standard measures like NMI and ARI.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Constant-baseline claim is false under standard null models: with n=3 and block sizes (2,1), expected ResMI is about 0.516, not 0, because the shared random pair couples X and Y.","rationale":"The paper's headline contribution is a clustering similarity measure that is claimed to be model-independent and to have zero expected value for independent clusterings. That property underpins Table II's Constant Baseline and Model Independence checkmarks and motivates the comparison with AMI/ARI. The counterexample shows the property is false in general: the expectation is positive, not zero. The subtlety is that even when f and g are independent, the indicators X and Y are linked through the random pair (Z1,Z2); this coupling is exactly what ResMI measures, but it also prevents the measure from having zero expectation merely because f and g are drawn independently. The reader's diagnosis is close, but the precise failure is not solely the permutation model's non-uniform per-pair probabilities; the mixture over random pairs can create dependence even when unconditional per-pair probabilities are uniform, so I mark agreement as partial. Other issues, such as the probable typo in qf|G, the lack of released code, and the qualitative real-data evaluation, are secondary. This concern is load-bearing but not necessarily fatal: the large-n experiments suggest the bias decays, so the authors could repair the paper by proving an asymptotic bound or restating the constant-baseline property as approximate for large n. That is consistent with the reader's CONDITIONAL verdict, so no adjustment is needed.","tokens_in":7354,"tokens_out":15089,"duration_ms":139719,"concrete_test":"Enumerate all 3x3 pairs of independent uniform partitions of n=3 with block sizes (2,1). For each pair, compute ResMI using the corrected conditional probability qf|G=(1/|G|)*sum_{(i,j) in G} 1{f(i)=f(j)} and the analogous expression for G^c, then report the mean. If the mean is about 0.516 rather than 0, the constant-baseline claim is refuted. As a robustness check under the paper's own large-n regime, run 10^5 Monte Carlo draws of independent uniform labelings on n=32 with c=4 clusters; if the mean ResMI is not within statistical error of 0, the exact property fails even when the empirical bias is small.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central theoretical advantage, stated in the Abstract, Sec. II-C, and Table II, is that ResMI satisfies the constant baseline property without chance correction. For this to hold when f and g are drawn independently, Eq. (7) would have to yield zero for essentially every pair of labelings in the support, since ResMI is nonnegative. That requires X=1{f(Z1)=f(Z2)} and Y=1{g(Z1)=g(Z2)} to be independent for each fixed f,g. Independence of f and g does not imply this: X and Y are functions of the same random pair (Z1,Z2), so their joint law is a mixture over pairs of independent Bernoulli variables, and the mixture can create dependence. Concrete counterexample: n=3, f and g independent uniform partitions with block sizes (2,1). If f=g (probability 1/3), ResMI=1; otherwise (probability 2/3), qf=qg=1/3, qf|G=0, qf|Gc=1/2, giving ResMI=(h2(1/3)-2/3)/h2(1/3)≈0.274. Expected ResMI ≈ 0.516, not 0. This is exactly the permutation model used to justify AMI and ARI, so the claimed advantage over chance-corrected measures fails where chance correction is standard. The large-n experiments (n=1024) only show the bias is numerically small for large n; the text's statement that experiments 'establish this result in generality' (Sec. II-C) is not a proof. A secondary issue: qf|G should be normalized by |G|, not binom(|G|,2); with |G|=1 the printed formula is undefined, likely affecting the reported implementation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ResMI, a clustering-similarity measure defined as the normalized mutual information between two co-membership indicators obtained by resampling a pair of objects without replacement. The authors claim that ResMI satisfies the constant baseline property without chance-correction terms, is model-independent, stays in [0,1], and avoids the cluster-count and symmetry biases of NMI, AMI, ARI, and RMI. These claims are supported by four synthetic experiments with n=1024 and by community-detection experiments on two contact-tracing networks.","tokens_in":7688,"tokens_out":12258,"duration_ms":191665,"significance":"If the central claims were correct, ResMI would be a practically attractive measure: it is interpretable in information-theoretic terms, bounded, and free of a user-specified null model. The paper also has useful experimental scaffolding, including comparisons with standard libraries and real network data. However, the main theoretical advantage advertised in the abstract and in Sec. II-C—exact constant baseline without chance correction—is not proven and is in fact false under the standard permutation model. The empirical results only show small finite-sample bias at n=1024 for one generative model. As a result, the paper's principal contribution, as stated, is not established.","major_comments":[{"comment":"The constant baseline claim is load-bearing and is false under the standard permutation model. In Eq. (7), X=1{f(Z1)=f(Z2)} and Y=1{g(Z1)=g(Z2)} are functions of the same random pair (Z1,Z2); independence of the random labelings f and g does not make X and Y independent. Concretely, for n=3, fix f to be the partition {12}|{3} and let g be drawn uniformly from the three partitions of [3] with block sizes (2,1). If g=f, ResMI=1 (probability 1/3). If g={13}|{2} or {23}|{1} (probability 2/3), then qf=qg=1/3, qf|G=0, and qf|Gc=1/2, so ResMI=(h_b(1/3)-2/3)/h_b(1/3)≈0.274. Hence E_g[ResMI(f,g)]≈0.516, not 0. Since ResMI is nonnegative, an exact constant baseline would require ResMI=0 on essentially the whole support; this counterexample shows it does not. The n=1024 simulations in Fig. 1(a) only show that the bias is numerically small in one large-n setting, and the sentence claiming that the experiments 'establish this result in generality' is not a proof.","section":"Sec. II-C, Eq. (7)"},{"comment":"The definition of qf|G is inconsistent with G being a set of pairs. The printed expression normalizes by binom(|G|,2), but the conditional probability is the average over the pairs in G, so the denominator should be |G|; the same correction applies to qf|Gc. As written, the formula is undefined when |G|=1 and otherwise scales the conditional probability incorrectly, which materially changes the value of ResMI and compromises the reproducibility of the experiments.","section":"Sec. II-C, definition of qf|G"},{"comment":"The experimental results are used to support the 'Constant Baseline ✓' entry for ResMI in Table II, but the experiments average over one particular generative process (random assignment to c clusters with n=1024) and do not establish an exact property of the measure. Table II should not list the constant baseline as an exact property unless a theorem is supplied; at most, the paper can claim low large-n baseline bias for the specific null models tested. The authors should either prove a precise asymptotic statement or remove the claim.","section":"Sec. III, Fig. 1 and Table II"}],"minor_comments":[{"comment":"The displayed formula for AMI is not the standard adjusted mutual information of Vinh et al., which corrects the mutual information I(f;g) rather than NMI; please reconcile the formula with the implementation in aricode used in the experiments.","section":"Sec. II-A, Eq. (3)"},{"comment":"The behavior of ResMI is not specified when both H(1{f(Z1)=f(Z2)}) and H(1{g(Z1)=g(Z2)}) are zero; the paper should state a convention for this degenerate case.","section":"Sec. II-C, Eq. (7)"},{"comment":"The claim that NMI and RMI 'fail to satisfy' the monotonicity property appears to be based on visual inspection of the plotted curves; a quantitative criterion or a statistical comparison would make the claim more precise.","section":"Sec. III, Fig. 1(c)"}],"recommendation":"reject","confidential_remarks":"The central theoretical advantage of the paper—exact constant baseline without chance correction—is false under the standard null model, and the finite-n counterexample in the report shows that no proof can repair the claim as stated. A substantially revised manuscript that reframes ResMI as an approximately unbiased large-n heuristic, with rigorous asymptotics and corrected formulas, could be reconsidered, but that would be a different contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the one thing you should know: the paper introduces a new clustering similarity measure, ResMI, defined as the normalized mutual information between two Bernoulli variables indicating whether a random pair of objects shares a label under each clustering. This is a clean, novel combination of pair counting and information theory, and it does fix the cluster-count bias that NMI has. The synthetic experiments are clearly described and the qualitative real-data analysis is reasonable.\n\nThe soft spot is the central theoretical claim. The abstract and Section II-C say ResMI satisfies the constant baseline property without chance correction. That only holds if the two Bernoulli variables are independent, which happens when labels are assigned to objects independently and uniformly. Under the permutation model used by AMI and ARI—where cluster sizes are fixed and labels are permuted—the shared random pair couples the two indicators. A concrete counterexample: for n=3 with a (2,1) partition, two independent uniform such partitions give expected ResMI ≈ 0.516, not 0. The paper's statement that \"experimental results establish this result in generality\" is not a proof; the n=1024 experiments only show the bias is numerically small in that setting.\n\nThere is also a typo in the definition of qf|G: it divides by C(|G|,2), but |G| counts pairs, so the correct normalizer is |G| (and C(1,2)=0, so the formula is undefined for the common case where a cluster has one pair). No code is provided, which makes the exact implementation hard to verify.\n\nAll that said, the core idea may still be useful. The measure behaves sensibly in the experiments and avoids the symmetry bias that hits ARI. But the paper overclaims the theoretical property. With a corrected definition and a clearly stated null model—one where the constant baseline actually holds, or a revised claim about approximate behavior—it could be a fine contribution.\n\nBottom line: worth sending to a serious referee, but the referee should be told to check the null model carefully and demand a fix to the formula. I probably wouldn't cite it in its current form.","headline":"ResMI is a genuinely new measure, but its flagship claim—constant baseline without chance correction—only holds for a special null model and fails under the standard permutation model.","tokens_in":8164,"tokens_out":3849,"would_cite":false,"duration_ms":32903,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62H30"],"pacs":[],"model":"deepseek-v4-flash","headline":"Resampled mutual information gives clustering comparisons a zero baseline without correction terms.","keywords":["clustering similarity","mutual information","constant baseline property","chance correction","community detection","pair counting","model independence","adjusted rand index"],"falsifier":"Compute the expected ResMI between two random partitions of $n=3$ objects with cluster sizes $2$ and $1$, drawn uniformly from all fixed-size partitions; the expected value is about $0.515$, not $0$, which would directly contradict the claim that the constant baseline holds in generality.","tokens_in":7144,"feed_emoji":"🧩","tokens_out":9069,"duration_ms":79739,"temperature":0.7,"pith_summary":"This paper introduces resampled mutual information (ResMI), a measure for comparing two clusterings, and argues that it solves a known defect of standard scores: the expected similarity between two independent clusterings should be zero, yet normalized mutual information (NMI) and the Rand index (RI) are not zero on average. ResMI instead draws two objects at random and records, for each labeling, whether the two objects land in the same cluster, then takes the normalized mutual information between those two Bernoulli indicators. The paper's central claim is that this resampling step gives the constant baseline property automatically, without a chance-correction term, so ResMI remains bounded in $[0,1]$, model-independent, and interpretable as shared information. Synthetic experiments and two real contact-tracing networks are used to show that ResMI avoids the cluster-count bias of NMI and the asymmetric-cluster bias of the adjusted Rand index.","feed_headline":"Zero-baseline clustering score skips chance correction","feed_subtitle":"By comparing random pairs instead of labels, independent clusterings score exactly zero.","key_machinery":"The machinery is the replacement of the labels themselves by pairwise agreement events. Let $G=\\{(i,j):g(i)=g(j), i>j\\}$; with $q_f$ the co-clustering rate under $f$, $q_g$ under $g$, and $q_{f|G}$, $q_{f|G^c}$ the conditional rates inside and outside $G$, the numerator of ResMI is $$I_{\\mathrm{ResMI}}(f;g)=h_b(q_f)-\\left[q_g h_b(q_{f|G})+(1-q_g)h_b(q_{f|G^c})\\right],$$ where $h_b$ is the binary entropy function. This identity turns the pair-counting intuition of the Rand index into an information-theoretic quantity and makes the mutual information depend only on four co-clustering probabilities rather than on the full contingency table. The same object delivers the constant baseline property: under independent random labelings of the uniform-label kind, the expected value of this expression is zero, so no chance-correction term is required.","core_discovery":"The central claim is that the right information-theoretic object for comparing clusterings is the mutual information between pair-co-clustering indicators, not between label assignments. Formally, for labelings $f,g$ and a uniformly random pair $(Z_1,Z_2)$ drawn without replacement, ResMI is $$\\mathrm{ResMI}(f,g)=\\frac{I(\\mathbf{1}\\{f(Z_1)=f(Z_2)\\};\\mathbf{1}\\{g(Z_1)=g(Z_2)\\})}{\\frac{1}{2}\\left(H(\\mathbf{1}\\{f(Z_1)=f(Z_2)\\})+H(\\mathbf{1}\\{g(Z_1)=g(Z_2)\\})\\right)}.$$ Because the two Bernoulli variables have probabilities $q_f$ and $q_g$ equal to the empirical co-clustering rates of the two labelings, the trivial labelings (all objects in one cluster, or each object in its own cluster) contribute zero information, and the author reports that the expected value under independently drawn random labelings is zero. This is exactly the constant baseline property that NMI and RI lack, achieved without subtracting an expected-value adjustment term. On this basis the paper claims ResMI is free of cluster-count bias, free of the symmetry bias that distorts ARI, and more conservative than NMI and RMI when clusterings depart from a ground truth.","pith_inferences":["An extension the paper leaves implicit is that the constant-baseline claim is tied to the uniform independent-label null model; deriving the closed-form expectation of ResMI under the fixed-cluster-size permutation model would delimit exactly when the baseline holds, and the $n=3$, sizes $(2,1)$ case suggests it does not hold there.","The pair-event formulation could transfer to overlapping or hierarchical clusterings by redefining the event 'same cluster' as 'share at least one community' or 'belong to a common nested block'; the paper does not test these variants.","Because ResMI depends only on four co-clustering probabilities, it may be computable in near-linear or quadratic time from the contingency table, making it a plausible direct objective for clustering algorithms; the paper mentions this direction but does not implement it."],"forward_implications":["ResMI can be used as a standalone, bounded clustering-similarity score without choosing a chance-correction model, removing a source of ambiguity documented for AMI and ARI.","It avoids NMI's tendency to reward labelings with many clusters and ARI's distortion under asymmetric cluster-size distributions, so rankings of competing clusterings can change in settings where those biases operate.","In the second contact-tracing network, ResMI agrees with AMI and ARI in selecting $c=10$ communities, where RMI chooses too few and NMI too many; if this pattern generalizes, ResMI is a more reliable guide for choosing the number of communities.","Because the measure is defined entirely through mutual information and entropy, it remains interpretable as shared information between two random variables, which chance-corrected scores lose."],"supporting_citations":[{"why":"Defines adjusted mutual information, the chance-corrected baseline that ResMI aims to match without a correction term.","marker":"[1]"},{"why":"Surveys NMI variants and their normalization and chance-correction properties; the comparison target and source of cluster-count bias.","marker":"[2]"},{"why":"Defines reduced mutual information (RMI), a compared measure whose correction term induces negative values.","marker":"[3]"},{"why":"Supplies the synthetic experimental setup for testing constant baseline and cluster-count bias.","marker":"[4]"},{"why":"Shows why chance-correction terms depend on the assumed random model, motivating ResMI's model-independence requirement.","marker":"[7]"},{"why":"Documents the adjusted Rand index's bias for asymmetric cluster sizes, the failure case ResMI is tested against.","marker":"[8]"},{"why":"Provides the first real contact-tracing network with ground-truth department labels used to evaluate ResMI.","marker":"[10]"},{"why":"Provides the second contact-tracing network, where the measures disagree on the optimal number of communities.","marker":"[11]"},{"why":"Supplies the SCORE+ spectral clustering algorithm used to generate estimated community labels for the real-data comparison.","marker":"[17]"}],"fun_headline_variants":["Resampled pairs reset clustering similarity baseline","Cluster score with constant zero baseline, no correction","ResMI pairs up clusterings, skips chance adjustment","Pair co-clustering information defines bias-free metric","Zero baseline for clustering from resampled pair info"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The zero baseline depends on the two Bernoulli co-clustering indicators being independent when labelings are generated at random, which holds for independent uniform label assignment but fails when cluster sizes are fixed as in the permutation model.","fun_headline_variants_meta":{"raw":{"variants":["Resampled pairs reset clustering similarity baseline","Cluster score with constant zero baseline, no correction","ResMI pairs up clusterings, skips chance adjustment","Pair co-clustering information defines bias-free metric","Zero baseline for clustering from resampled pair info"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1517,"prompt_tokens":911,"completion_tokens":606,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":533}},"tokens_in":527,"tokens_out":606,"duration_ms":6021,"temperature":1.0,"reasoning_tokens":533,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:50:16.328205+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the expected ResMI between two random partitions of $n=3$ objects with cluster sizes $2$ and $1$, drawn uniformly from all fixed-size partitions; the expected value is about $0.515$, not $0$, which would directly contradict the claim that the constant baseline holds in generality.","supporting_citations":[{"cited_title":"Information theoretic measures for clusterings comparison: Is a correction for chance necessary?","cited_arxiv_id":null,"evidence_quote":"Defines adjusted mutual information, the chance-corrected baseline that ResMI aims to match without a correction term."},{"cited_title":"Information theoretic measures for clusterings comparison: Vari- ants, properties, normalization and correction for chance,","cited_arxiv_id":null,"evidence_quote":"Surveys NMI variants and their normalization and chance-correction properties; the comparison target and source of cluster-count bias."},{"cited_title":"Improved mutual infor- mation measure for clustering, classification, and community detection,","cited_arxiv_id":null,"evidence_quote":"Defines reduced mutual information (RMI), a compared measure whose correction term induces negative values."},{"cited_title":"Element-centric clustering comparison unifies overlaps and hierarchy,","cited_arxiv_id":null,"evidence_quote":"Supplies the synthetic experimental setup for testing constant baseline and cluster-count bias."},{"cited_title":"The impact of random models on clustering similarity,","cited_arxiv_id":null,"evidence_quote":"Shows why chance-correction terms depend on the assumed random model, motivating ResMI's model-independence requirement."},{"cited_title":"Understanding the adjusted rand index and other partition comparison indices based on counting object pairs,","cited_arxiv_id":null,"evidence_quote":"Documents the adjusted Rand index's bias for asymmetric cluster sizes, the failure case ResMI is tested against."},{"cited_title":"Data on face-to-face contacts in an office building suggest a low-cost vaccination strategy based on community linkers,","cited_arxiv_id":null,"evidence_quote":"Provides the first real contact-tracing network with ground-truth department labels used to evaluate ResMI."},{"cited_title":"Can co-location be used as a proxy for face- to-face contacts?","cited_arxiv_id":null,"evidence_quote":"Provides the second contact-tracing network, where the measures disagree on the optimal number of communities."},{"cited_title":"Improvements on score, especially for weak signals,","cited_arxiv_id":null,"evidence_quote":"Supplies the SCORE+ spectral clustering algorithm used to generate estimated community labels for the real-data comparison."}],"review_version":1}