{"id":"cc3535bb-05a4-4180-b153-e0f23a870cbc","arxiv_id":"2607.23998","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A truncated harmonic mean estimator computes stochastic-block-model marginal likelihoods from MCMC samples, achieving simulation-consistency at fixed data size and handling label switching efficiently.","lead":"A new method estimates the statistical evidence for different numbers of clusters in network data by averaging over Markov chain Monte Carlo samples, and it converges as the simulation grows. The authors test it on tiny synthetic networks where the true answer is known, then apply it to a large X/Twitter dataset from the 2023 COP28 climate conference, concluding that the data support 12 clusters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best-of-10 MAP chain selection in the COP28 pipeline violates the posterior-sample assumption of Theorem 1, so the real-data marginal likelihood estimates are not covered by the consistency guarantee.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the best-of-10 MAP selection in Supplement C.1 means the MCMC sample used for the COP28 analysis is not a draw from the posterior p(C|G,Y), so Theorem 1 does not apply to that analysis. This is concrete, located in the text, and directly relevant to the paper's headline application. A simple simulation with known Z can determine whether the selection bias is numerically important. The theoretical estimator itself may be sound when a single converged chain is used, so I do not recommend changing the reader's CONDITIONAL verdict: the paper should be revised to either remove the chain-selection step, justify it as an approximation with quantitative bounds, or rerun the COP28 analysis on a proper posterior sample.","tokens_in":35228,"tokens_out":13033,"duration_ms":140502,"concrete_test":"Use the small-n exact-Z setting from Section 4 (n=10, G=2). For each simulated dataset, run 10 independent Gibbs chains of length T (e.g., 1,000, 2,000, 5,000, 10,000), retain the chain with maximal MAP as in Supplement C.1, compute THAMES on that chain, and compare log(Zhat) to log(Z) over at least 100 datasets. If the average bias does not shrink toward zero as T increases, or if it is significantly larger than the single-chain THAMES Monte Carlo error, the best-of-10 selection invalidates the consistency guarantee in the implemented pipeline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that THAMES is simulation-consistent (Theorem 1) when C^(1),...,C^(T) are ergodic draws from p(C|G,Y). In the real-data pipeline (Supplement C.1), 10 independent Gibbs samplers are run for each G and only the chain with the highest MAP value is retained. The retained draws are therefore not marginally distributed as p(C|G,Y): the chain index is informative about the maximized unnormalized posterior. This can bias the RIS average in Eq. (4), because selecting high-MAP states preferentially discards draws with large reciprocal-weight terms, and the bias may persist for finite T. The reported COP28 results (G=12, Table 2) and the sociological interpretation rest on estimates computed from such selected chains. This is not a theoretical flaw in THAMES for a single converged chain, but it is an unaddressed violation of the theorem in the application that motivates the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes THAMES, a truncated harmonic mean estimator for the marginal likelihood of stochastic block models. The estimator is defined as a symmetrized reciprocal importance sampling average over the second half of an MCMC sample, with a truncation set built from the first half using a variational approximation. The authors claim that THAMES is simulation-consistent and asymptotically normal under standard MCMC conditions, invariant to label switching, computationally efficient for large numbers of clusters, and applicable to large networks. The theoretical argument is based on the standard RIS identity (Theorem 1); the truncation set is motivated by an HPD optimality result (Theorem 2); an efficient summation over permutations is given (Theorem 3); and an empty-cluster correction is adapted from Nobile's formula (Theorem 4). The method is tested on n=10, G=2 simulations where the true marginal likelihood is available, and then applied to a COP28 X (Twitter) dataset, yielding G=12 as the preferred number of clusters.","tokens_in":35487,"tokens_out":8296,"duration_ms":91982,"significance":"If the claims hold, this would be a useful contribution: a marginal likelihood estimator for SBMs that does not evaluate the intractable likelihood and that is consistent as the number of MCMC simulations grows, not as the network size grows. The paper also provides a practical device for label-switching invariance and a computational shortcut that avoids the G! factorial sum. The explicit proofs in the supplement, the availability of an R package, and the extension to latent block models are strengths. However, the simulation evidence is limited to a very small setting (n=10, G=2), and the real-data application uses a chain-selection protocol that is not covered by the stated theory. The claimed asymptotic normality also lacks a theorem covering the data-dependent choice of the truncation parameter. These gaps do not invalidate the core RIS idea, but they need to be addressed before the paper's stronger claims can be accepted.","major_comments":[{"comment":"The real-data pipeline runs 10 independent Gibbs samplers per G and retains only the chain with the highest MAP value. Theorem 1 requires C^(1),...,C^(T) to be an ergodic Markov chain with limiting distribution p(C|G,Y). A chain selected by maximizing the unnormalized posterior is not marginally distributed as p(C|G,Y). Because the RIS weight in Eq. (4) is the reciprocal of p(C|G)p(Y|G,C), the selection is informative about the weights: the retained chain is tilted toward large values of this product, so the average of its reciprocals is biased downward as an estimate of 1/Z(G), and log Z(G) is biased upward. The bias can vary with G and is not covered by the consistency guarantee. The COP28 estimates and the choice G=12 therefore rest on an unaddressed violation of the theorem. A single converged chain or a proper combination of multiple chains is needed.","section":"Supplement C.1, Table 2"},{"comment":"The truncation parameter alpha is chosen by minimizing the estimated variance of THAMES computed on the same second-half sample used for the point estimate. Theorem 1 and its proof in Supplement A.1 apply to a fixed importance function h, equivalently a fixed truncation set A. When A is selected on the data used for the estimate, the estimator is not a standard RIS estimator, and no theorem is provided for the resulting procedure. The remark in Section 3.1 that 'for a fixed value of alpha' consistency holds does not cover the data-dependent choice. The abstract's asymptotic-normality claim therefore lacks support. The authors should either prove a uniform-in-alpha result for the grid selection, or choose alpha on an independent pilot sample and state the additional conditions needed for the CLT.","section":"Section 3.1, Supplement B.1"},{"comment":"The paper claims that THAMES allows approximate confidence intervals, but Table 2 reports rounded log-marginal-likelihood estimates with no standard errors. The difference between G=12 and G=13 is only 7 on the log scale. Without Monte Carlo uncertainty quantification, it is not possible to judge whether this difference is meaningful, especially since the estimates are not covered by Theorem 1 because of the chain-selection issue in Supplement C.1. The model-selection conclusion and the subsequent sociological interpretation depend on this table. At minimum, the authors should report standard errors or intervals for all G and discuss the sensitivity of the argmax to Monte Carlo error.","section":"Section 5, Table 2"}],"minor_comments":[{"comment":"The empty-cluster correction via Theorem 4 requires p0(G)>0. If the MCMC sample contains no empty clusters, the empirical estimator of p0(G) is 0 and Eq. (12) is undefined. The paper should clarify the procedure in that case, even if the correction is only applied when an empty cluster is observed.","section":"Section 3.3 / Supplement A.6"},{"comment":"The cross-reference 'Equation (25)' in Remark 2 appears to refer to Equation (11) in the main text; similarly, 'Equation (26)' in Section 3.3 should be Equation (12). Please correct the equation numbering in the main text.","section":"Section 3.2, Remark 2"},{"comment":"The simulation study is limited to n=10, G=2, where the true marginal likelihood has only 1024 terms. This demonstrates the estimator on a toy setting but does not validate the computational or statistical behavior for larger n and G, which is where the method is claimed to be useful. A larger simulation, even without an analytic gold standard, would strengthen the empirical case.","section":"Section 4"},{"comment":"There are several typos: 'connecction' should be 'connection'; in Supplement C.3, 'alocation' should be 'allocation'; 'the Latent dirichlet alocation (LDA) model' is redundant. These do not affect content but should be cleaned up.","section":"Supplement C.2"},{"comment":"The statement that the size of Omega 'is usually equal to one in our experience' is informal. Since this is the basis for the claimed efficiency for arbitrarily large G, the authors should either provide a theoretical bound on |Omega| under explicit posterior-concentration conditions or at least report the distribution of |Omega| in the simulations and the COP28 application.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The core RIS identity is sound, and the paper contains useful ideas. The main concern is that the application section uses a best-of-10 MAP chain selection that violates the theorem, and the asymptotic-normality claim is not supported for the data-dependent truncation parameter. Both are fixable within the scope of a revision. The editor may wish to ask for a formal treatment of data-dependent truncation and a re-analysis of the COP28 data with a statistically valid MCMC protocol, or at least a clear statement that those estimates are exploratory and not covered by Theorem 1."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good to talk. The short version: the THAMES estimator is a genuine and potentially useful contribution, and the consistency proof is standard and correct for a fixed truncation set. But the real-data section selects the best of 10 Gibbs chains by MAP, which breaks the posterior-sample assumption; and the asymptotics ignore the data-dependent choice of alpha. Both are fixable, but they need attention.\n\nWhat is actually new: the sample-splitting truncation set, the symmetrization over label permutations via a lexicographic ordering that avoids the G! explosion, and the empty-cluster recursion via Nobile's formula. These are real ideas, and the computational trick is neat. The paper is honest about the strong assumptions behind the optimality of the HPD region (Remark 1), and the proofs in the supplement are reasonably clear.\n\nThe soft spots, in proportion: the most serious is in Supplement C.1. Running 10 independent samplers and keeping the chain with the highest MAP biases the retained draws toward high-likelihood modes. Theorem 1 requires draws from the posterior; the COP28 estimates are therefore not covered by the consistency guarantee. This is not a flaw in the estimator, but it means the headline application results are not backed by the theory. Second, the asymptotic normality claim holds only for fixed alpha; the variance-minimization of alpha on the same second-half sample makes the estimator data-dependent in a way that the proof does not address. This is likely fixable with a separate tuning set or a uniformity argument, but as written it's a gap. Third, the simulations are thin: n=10, G=2 only, with no evidence at larger scales where the computational trick matters. Fourth, the novelty claim is slightly overblown—McDaid et al. (2012) already did something similar for a collapsed SBM with random G. The specific THAMES variant is new, but the \"first simulation-consistent\" phrasing needs a qualifier. Finally, the GitHub link is missing, which hurts reproducibility.\n\nWho is this for: anyone working on Bayesian model selection for block models, especially in network science. The method is worth knowing even if the current application is not fully valid. I would send this to peer review—the core is sound and the issues are addressable—but the authors need to deal with the chain selection before the real-data results can be trusted.","headline":"A clever and potentially useful RIS estimator for SBM marginal likelihoods, but the real-data analysis violates its own consistency guarantee by selecting the best of 10 chains, and the asymptotics ignore data-dependent tuning.","tokens_in":35952,"tokens_out":3658,"would_cite":true,"duration_ms":40492,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","62H30","62M05"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces the truncated harmonic mean estimator (THAMES), a simulation-consistent estimator of the marginal likelihood for stochastic block models that avoids evaluating the likelihood and handles label switching.","keywords":["marginal likelihood","stochastic block model","model selection","harmonic mean estimator","reciprocal importance sampling","label switching","MCMC","Bayesian clustering"],"falsifier":"Simulate a network with n=10 and G=2 (where Z(G) can be computed exactly by summing all 1024 configurations), run Gibbs sampling with the same chain-selection protocol used on the COP28 data, and compute THAMES on the selected chain as T grows. If the estimate does not converge to the true Z(G), or if it systematically differs from THAMES computed on the full unselected sample, the chain-selection step is biasing the estimator.","tokens_in":35123,"feed_emoji":"📊","tokens_out":4863,"duration_ms":45506,"temperature":0.7,"pith_summary":"The paper introduces THAMES, the first estimator of the marginal likelihood for stochastic block models that is simulation-consistent: as the MCMC sample size grows, it converges to the true marginal likelihood even when the network size is fixed. This matters because choosing the number of clusters in a network is a Bayesian model selection problem, and existing variational or asymptotic approximations lack finite-sample consistency guarantees. The estimator uses only the MCMC draws of the cluster allocation vectors, exploiting the analytic collapsed prior and collapsed likelihood, so it never evaluates the intractable SBM likelihood. It is asymptotically normal and symmetric under label permutations, and a computational shortcut keeps the cost manageable even when the number of clusters is large. On a real COP28 Twitter/X network, THAMES selects 12 clusters and exposes a sharp core-periphery structure.","feed_headline":"Cluster counts in networks become consistently estimable","feed_subtitle":"The THAMES estimator converges to the true marginal likelihood as simulations grow, even for small networks.","key_machinery":"The load-bearing identity is the reciprocal importance sampling representation: for any density h on allocation vectors, the average of h(C^(t))/(p(Y|G,C^(t))p(C^(t)|G)) over posterior draws converges to 1/Z(G). THAMES chooses h as a symmetrized uniform density on a truncation set B = E ∩ H_alpha, where E is a Cartesian product of node-level candidate sets from the first half of the MCMC output and H_alpha is a highest-posterior-density region; this choice makes the estimator label-switching invariant and approximately variance-optimal. The analytic forms of the collapsed prior and collapsed likelihood supply the unnormalized posterior needed in the denominator. A lexicographic ordering of t","core_discovery":"The central claim is that the marginal likelihood Z(G) of a stochastic block model can be estimated consistently from a Gibbs sample of allocation vectors alone. The estimator is a reciprocal importance sampling estimator whose proposal h is a mixture of uniform distributions over the label-permuted copies of a truncation set A; the truncation set is chosen as the intersection of a Cartesian product of per-node candidate allocations (from the first half of the sample) with a highest-posterior-density region, which the paper shows minimizes variance among symmetric sets under mild conditions. Because the collapsed distributions p(C|G) and p(Y|G,C) are known analytically, each term is computab","pith_inferences":["A testable extension is to compare THAMES computed on the single highest-MAP chain versus on all chains pooled: if the chain-selection protocol biases the posterior sample, the two estimates should drift apart as T grows, and the pooled version should be closer to the truth in settings with known Z(G).","The variance-optimality of the HPD truncation set presumes no posterior ties; in datasets with near-tied allocation probabilities, a randomized tie-breaking or a smoothed truncation set might change the variance ranking, an effect the paper does not explore.","The empty-cluster recursion suggests a direct route to estimating the marginal likelihood for models with random G by chaining THAMES estimates across G, which would make trans-dimensional model selection consistent as well.","The core-periphery pattern found in the COP28 data implies a diagnostic: when a network's posterior mass is concentrated on a few hub-like accounts, marginal-likelihood-based cluster counts may be highly sensitive to the inclusion or exclusion of those hubs; the method's behavior on such sparse, hub-dominated networks deserves specific simulation study."],"forward_implications":["Bayesian model selection for the number of clusters in a network becomes simulation-consistent even for small networks, with confidence intervals available from the asymptotic normality.","Any block model with analytically collapsed allocation distributions — including the latent block model — can use the same estimator, extending consistent evidence computation to co-clustering.","The COP28 analysis indicates that marginal-likelihood-based cluster choice can disagree sharply with criteria like ICL, yielding a less fragmented interpretation of the network's core-periphery structure.","The computational shortcut makes the estimator practical for networks with tens of thousands of nodes and cluster counts well beyond 15, where permutation-averaged alternatives are infeasible.","Because the estimator uses only the allocation vectors, it applies even when the likelihood itself is intractable, as long as Gibbs sampling is possible."],"fun_headline_variants":["Consistent marginal likelihoods for block models","THAMES: consistent evidence from Gibbs samples","Block models: fixed-size consistent likelihoods from MCMC","Counting clusters: a consistent estimator"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The consistency theorem assumes the MCMC allocation draws follow the posterior p(C|G,Y); in the real-data analysis the authors run ten Gibbs chains and keep only the one with the highest MAP value, which can bias the retained sample away from the posterior and break the guarantee.","fun_headline_variants_meta":{"raw":{"variants":["Consistent marginal likelihoods for block models","THAMES: consistent evidence from Gibbs samples","Block models: fixed-size consistent likelihoods from MCMC","Counting clusters: a consistent estimator"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1264,"prompt_tokens":614,"completion_tokens":650,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":358,"completion_tokens_details":{"reasoning_tokens":594}},"tokens_in":358,"tokens_out":650,"duration_ms":7045,"temperature":1.0,"reasoning_tokens":594,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T23:19:20.030164+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate a network with n=10 and G=2 (where Z(G) can be computed exactly by summing all 1024 configurations), run Gibbs sampling with the same chain-selection protocol used on the COP28 data, and compute THAMES on the selected chain as T grows. If the estimate does not converge to the true Z(G), or if it systematically differs from THAMES computed on the full unselected sample, the chain-selection step is biasing the estimator.","supporting_citations":[],"review_version":1}