{"id":"b10a7e15-e7b4-409d-95f2-f7249211302c","arxiv_id":"2506.00245","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SNNE estimates LLM uncertainty from pairwise semantic similarities of sampled answers using a log-sum-exp aggregation, and it generalizes semantic entropy as a special case.","lead":"This paper introduces SNNE, a new way to measure how unsure a large language model is about its own answers by comparing how similar its sampled responses are to each other. It reports better hallucination detection than semantic entropy across question answering, summarization, and translation tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set hyperparameter selection (τ and ROUGE-L) may inflate reported gains; a held-out validation split is needed before the central claim is credible.","rationale":"The reader's verdict (CONDITIONAL) already flags test-set hyperparameter tuning as a main weakness, but the reader's weakest_assumption focused on ROUGE-L's semantic validity and the 10-sample generation protocol. I agree that those are relevant, but the single most load-bearing concern is the test-set selection of τ and ROUGE-L, because it affects every reported comparison and directly undermines the out-of-sample claim. The reader's rationale does mention this, so there is partial agreement. My proposed concrete test—a validation split or a pre-registered default with independent evaluation—would settle whether the reported gains are real or an artifact of selection. The theoretical results are correct but do not address this empirical issue. Therefore the verdict should remain CONDITIONAL: the method is promising, but the central claim needs a fair evaluation before acceptance.","tokens_in":12641,"tokens_out":3932,"duration_ms":38857,"concrete_test":"Re-run the evaluation with a held-out validation split. For each dataset, reserve 20% of questions as a validation set; select τ (from {0.1,1,10,100}) and the similarity function (ROUGE-L vs. entail vs. embed) using only the validation set; then compute AUROC/PRR on the remaining 80% test set. If SNNE/WSNNE still outperform SE by a meaningful margin (e.g., average AUROC gap ≥ 0.01 and PRR gap ≥ 0.02) consistently across tasks, the concern is resolved. A complementary test: fix τ=1 and ROUGE-L a priori (the paper's stated default) and evaluate on an independent set of long one-sentence outputs from a model or dataset not used in the paper; if the improvement persists, the central claim holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim—that SNNE/WSNNE 'consistently outperform' semantic entropy across QA, summarization, and translation—rests on a test-set selection procedure for the two hyperparameters that most directly control the estimator. In Appendix B ('Hyperparameter setting'), the authors state: 'We select the best scale factor τ in Eq 3 and 4 from {0.1,1,10,100}. For similarity function f, we use ROUGE-L score.' They then report all main results (Figures 2–3, Tables 2–4) on the same datasets used to make that selection. Table 5 ('Effect of similarity function') and Table 6 ('Effect of scale factor') are computed on the same test tasks, so ROUGE-L and τ are effectively chosen on the test set. This is not a held-out evaluation: the reported differences, e.g., SNNE vs. DSE on summarization (PRR 0.26 vs. 0.14), could reflect overfitting to the specific datasets and generation conditions rather than a general advantage. Because every quantitative comparison in the paper inherits this selection, the 'consistently outperforms' claim is not yet supported as an out-of-sample statement. The theoretical results (Theorems 4.1, 4.2) are sound, but they apply to contrived similarity functions, not to the ROUGE-L f used in practice, so they do not mitigate this concern. The issue is addressable with a proper validation split, which is why this is a CONDITIONAL-accept-level concern rather than a rejection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SNNE, a black-box uncertainty quantification method for LLMs that computes, for each of n sampled model outputs, a log-sum-exp of pairwise semantic similarities with all other outputs, scaled by a temperature τ. A white-box variant WSNNE reweights the outer sum by normalized sequence probabilities. The authors argue that SE and DSE ignore intra- and inter-cluster similarity and therefore degrade on longer one-sentence generations, whereas SNNE captures both. They prove (Theorems 4.1–4.2) that DSE and SE are recovered as special cases under specially chosen similarity functions, and they report experiments on QA, summarization, and translation with several LLMs, comparing against a wide range of baselines and claiming consistent improvements.","tokens_in":12895,"tokens_out":7978,"duration_ms":68673,"significance":"If the empirical gains are robust, SNNE is a simple, intuitive, and computationally efficient (O(n^2)) alternative to SE for long-generation hallucination detection, and the white-box extension adds flexibility. The theoretical unification of SE/DSE as special cases is elegant, and the code is made available. However, the current evaluation does not yet establish the out-of-sample superiority claim because key hyperparameters are selected on the test sets and the main tables lack error bars. The conceptual theoretical results, while sound, do not speak to the specific choice of ROUGE-L as the similarity function.","major_comments":[{"comment":"Test-set hyperparameter selection undermines the central 'consistently outperforms' claim. In Appendix B, the authors state 'We select the best scale factor τ in Eq 3 and 4 from {0.1,1,10,100}. For similarity function f, we use ROUGE-L score.' All main results (Tables 2–4, Figures 2–3) are computed on the same datasets used for this selection. Table 5 shows that the choice of f is consequential: on summarization, ROUGE-L gives PRR 0.26 versus 0.22 for entail and embed; on translation, 0.63 versus 0.60 and 0.62. Thus the reported gains could be partly an artifact of choosing the similarity function that best matches the test tasks. Please provide a held-out validation split for f and τ selection (or, failing that, report results for all candidate choices for every dataset) before the superiority claim can be accepted as an out-of-sample statement.","section":"Appendix B, Appendix C.2, Tables 5–6"},{"comment":"Main results are reported without error bars despite three repetitions. Appendix B notes 'We conduct each experiment three times using NVIDIA RTX A6000 GPUs,' but Tables 2–4 give only point estimates. Without standard deviations or confidence intervals, differences such as SNNE 0.84 vs. SE 0.80 on Phi-3 QA or SNNE 0.26 vs. SE 0.20 on summarization cannot be assessed for statistical significance. Please report mean ± std (or CIs) for all main tables, as is already done in Appendix C.2.","section":"Tables 2–4"},{"comment":"The evaluation uses ROUGE-L both as the similarity function inside SNNE and as the correctness measure for the PRR results on summarization and translation. This dual use creates a potential positive bias for SNNE: its uncertainty score is directly tied to the same lexical-overlap measure that defines correctness, whereas baselines based on entailment or embeddings are not so aligned. While the paper does report BERTScore as a second correctness measure, the same concern applies to the choice of f. Please discuss this potential circularity and, ideally, add a correctness measure independent of the similarity function (e.g., human-annotated correctness or an entailment-based label) to confirm the gains are not an artifact of metric alignment.","section":"Table 4, Appendix B"},{"comment":"The theoretical 'generalization' results do not support the empirical advantage claimed for ROUGE-L. Theorems 4.1 and 4.2 construct ad-hoc similarity functions (constant intra-cluster similarity with zero inter-cluster similarity, or a probability-weighted variant) under which SNNE/WSNNE algebraically reduce to DSE/SE. These functions are not the ROUGE-L similarity used in the experiments, so the theorems only demonstrate conceptual generality, not that ROUGE-L-based SNNE should outperform SE. The paper should avoid implying that the theorems explain the empirical gains; at most they position (W)SNNE as a flexible framework.","section":"Theorems 4.1–4.2, Section 4"}],"minor_comments":[{"comment":"There is an apparent typo: 'sample 10 answers at a high temperature (T= 10)' should likely read (T= 1.0), since Figure 4 and the text in Section 5.1 state that the generation setting is temperature 1.0 adopted from Farquhar et al. (2024).","section":"Appendix B"},{"comment":"The citation for Qwen2-7B-instruct points to 'Towards general text embeddings with multi-stage contrastive learning' (Li et al., 2023), which appears to be the wrong reference; please verify the correct source.","section":"Appendix C.2"},{"comment":"In the sentence 'The superior of ROUGE-L also explains the good performance of LexSim', 'superior' should be 'superiority'.","section":"Appendix C.2"},{"comment":"The right panel would benefit from error bars or a statement about whether the curve is averaged over multiple seeds, since the main text emphasizes 'consistently'.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The test-set selection issue is the main barrier; if the authors can provide a clean validation split or show that the choice of f and τ does not affect the ranking of methods (e.g., by reporting all candidate values for all datasets), the paper would be suitable for publication. The lack of error bars is also an easy fix. I would not recommend rejection because the method is simple, novel, and the theoretical unification is interesting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my take on arXiv:2506.00245. The core idea is simple and right: instead of clustering outputs then counting clusters, use pairwise similarities with a log-sum-exp aggregator. That gives you a cheap, black-box uncertainty score that naturally accounts for both intra-cluster spread and inter-cluster distance. The theoretical reduction to DSE/SE under degenerate similarity functions is a clean formal observation, and the white-box extension WSNNE is a natural weighting by token probabilities. On the empirical side, the method is tested on six models across QA, summarization, and translation, with code released. That's solid work and a useful addition to the LLM uncertainty toolbox.\n\nThe soft spots are real but not fatal. The main issue is the evaluation protocol: tau and the similarity function f are effectively chosen on the test sets (Appendix B says they select tau from {0.1,1,10,100}, and Table 5 compares ROUGE-L vs entail vs embed on the same tasks used for reporting). The main tables (2-4) report no error bars, so we can't see run-to-run variance. Table 6 does show tau barely matters across the range, which softens the overfitting worry; Table 5 shows ROUGE-L is clearly best on summarization but only ties on translation. Still, the \"consistently outperforms\" claim is an in-sample statement, not an out-of-sample one.\n\nTwo smaller issues. The theory uses similarity functions that are never used in the experiments; it's a nice algebraic rewrite but it doesn't tell us why ROUGE-L works. And for summarization and translation, the correctness metric is ROUGE-L while f is also ROUGE-L; that's not circular, but it's a potential source of favorable bias that deserves discussion. Also, the abstract says \"two recent LLMs\" while the experiments use six; minor inconsistency.\n\nAll of this is addressable with a clean validation split and error bars on the headline tables. The central idea is sound and the method is likely to be a useful complement to semantic entropy, especially for longer single-sentence outputs. The paper deserves a serious referee, not a desk reject, but the revision should be held to a proper evaluation protocol before the gains are taken at face value.\n\nRecommendation: conditional accept with a request for held-out validation and uncertainty reporting. I'd cite it once it's cleaned up.","headline":"Promising, simple uncertainty score that improves on semantic entropy for longer outputs, but test-set hyperparameter selection and missing error bars mean the gains need a held-out check before being taken at face value.","tokens_in":13460,"tokens_out":3740,"would_cite":true,"duration_ms":33956,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Measuring LLM uncertainty by aggregating pairwise similarities between sampled answers, rather than clustering them into semantic groups, detects hallucinations more accurately on long one-sentence outputs and provably generalizes…","keywords":["uncertainty quantification","hallucination detection","semantic entropy","nearest neighbor entropy","pairwise similarity","log-sum-exp","large language models","ROUGE-L"],"falsifier":"Construct a set of long one-sentence questions whose correct answers are phrased entirely differently from the model's sampled responses (high semantic equivalence, near-zero ROUGE-L); if SNNE then assigns high uncertainty to those correct answers, or its AUROC for hallucination detection falls below that of semantic entropy, the central claim fails.","tokens_in":12402,"feed_emoji":"🤖","tokens_out":11163,"duration_ms":97402,"temperature":0.7,"pith_summary":"Semantic entropy detects LLM hallucinations by clustering reworded answers and measuring uncertainty over clusters, but it degrades as models produce longer one-sentence outputs, where each answer tends to land in its own cluster and the entropy becomes uninformative. The paper introduces Semantic Nearest Neighbor Entropy (SNNE), which replaces clustering with a log-sum-exp aggregation of pairwise similarities between sampled answers, thereby capturing both the spread within a semantic group and the distance between groups. It proves that with a specific similarity function SNNE reduces exactly to discrete semantic entropy, and its white-box variant reduces to semantic entropy, so the new estimator strictly generalizes the old one. Across question answering, summarization, and machine translation on two recent LLMs, SNNE and its white-box counterpart report higher AUROC and PRR scores than semantic entropy and other baselines for flagging hallucinated outputs.","feed_headline":"Pairwise similarities beat semantic entropy for LLM uncertainty","feed_subtitle":"A log-sum-exp over answer similarities captures within- and between-cluster spread that clustering misses.","key_machinery":"The central object is the Semantic Nearest Neighbor Entropy (SNNE) of Equation 3, $\\mathrm{SNNE}(q) = -\\frac{1}{n}\\sum_{i=1}^{n}\\log\\sum_{j=1}^{n}\\exp(f(a_i,a_j|q)/\\tau)$, a nearest-neighbor entropy estimate in which the log-sum-exp acts as a smooth maximum so that outlier answers do not dominate the sum. The similarity function $f$ is plug-in: the experiments use ROUGE-L, with NLI entailment scores or sentence-embedding cosine similarity as alternatives, and the scale factor $\\tau$ controls how sharply close answers count. Theorems 4.1 and 4.2 show that particular choices of $f$ reduce SNNE to discrete semantic entropy and its white-box variant to semantic entropy, which is what licenses the claim that pairwise-similarity aggregation subsumes cluster-based semantic entropy.","core_discovery":"Hallucination in large language models can be flagged by estimating how uncertain the model is about its answer, and the standard uncertainty measure, semantic entropy, is built on a clustering step that groups semantically equivalent answers. As modern LLMs generate longer one-sentence responses, those answers increasingly fall into distinct clusters, so cluster-level entropy becomes nearly constant and loses its signal. The paper's central claim is that replacing clustering with pairwise similarity aggregation fixes this. Specifically, for each sampled answer SNNE computes the log-sum-exp of its similarity to all other sampled answers, averages over answers, and defines uncertainty as negative that average; WSNNE weights the same sum by normalized sequence probabilities in the white-box setting. Theorems show that when similarity is constant within a cluster and zero across clusters, SNNE equals discrete semantic entropy, and WSNNE equals semantic entropy, proving that the pairwise form strictly generalizes the cluster-based form. With ROUGE-L as the similarity function, the method reports higher AUROC on question answering and higher PRR on summarization and translation than semantic entropy and several baselines.","pith_inferences":["A unification of uncertainty quantification methods may follow: if pairwise-similarity aggregation subsumes semantic entropy, then future refinements to clustering-based entropy could be recast as particular similarity choices, giving a common language for comparing methods.","The heavy reliance on a single similarity function suggests an ensemble of lexical and embedding-based similarities might further stabilize the estimate on paraphrastic outputs; the paper tests each similarity alone, not their combination.","For multi-sentence outputs, applying SNNE at the sentence level and aggregating, along the lines LUQ aggregates atomic scores, is a natural next experiment that would test whether the pairwise principle extends beyond one-sentence generations."],"forward_implications":["Hallucination detection on long one-sentence outputs such as summaries and translations improves without needing access to the model's internal probabilities, using only a handful of sampled answers per question.","Because SNNE reduces to discrete semantic entropy and WSNNE to semantic entropy under specific similarity choices, pipelines already built on semantic entropy can be upgraded without changing their operational meaning.","The plug-in similarity function and temperature allow the estimator to be adapted to other output formats, such as code or mathematical expressions, by swapping the similarity function $f$.","The log-sum-exp aggregation makes SNNE less sensitive to outlier answers than graph-based methods that sum pairwise similarities, which the paper argues improves reliability of the entropy estimate."],"supporting_citations":[{"why":"Defines semantic entropy and discrete semantic entropy, the baselines the paper improves on, and supplies the generation protocol (10 samples at temperature 10) used in the experiments.","marker":"Farquhar et al., 2024"},{"why":"Provides the nearest-neighbor entropy estimation background that motivates the log-sum-exp aggregation in Eq. 3.","marker":"Beirlant et al., 1997"},{"why":"Introduces ROUGE-L, the similarity function used in the main SNNE experiments.","marker":"Lin, 2004"},{"why":"Supplies the LexSim baseline and the lexical-overlap similarity idea that ROUGE-L operationalizes.","marker":"Fomicheva et al., 2020"},{"why":"Presents graph-based black-box uncertainty methods (Deg, SumEigv, Eccen) that serve as baselines and share the use of pairwise semantic similarities.","marker":"Lin et al., 2024"},{"why":"Introduces KLE, a kernel-based entropy that also generalizes semantic entropy and serves as a white-box and black-box baseline.","marker":"Nikitin et al., 2024"},{"why":"Defines LUQ-Pair, which SNNE asymptotically recovers as the temperature parameter tends to zero and whose atomic scoring the paper suggests its method could replace.","marker":"Zhang et al., 2024"}],"fun_headline_variants":["Pairwise similarity beats semantic entropy for LLM uncertainty","Log-sum-exp over answer similarities flags LLM hallucinations","SNNE: a pairwise uncertainty measure that generalizes semantic entropy","From cluster centroids to pairwise similarities: better LLM uncertainty","One simple formula replaces cluster entropy for LLM uncertainty"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that ROUGE-L overlap between sampled answers faithfully reflects semantic similarity for long one-sentence outputs, so low overlap signals genuine uncertainty rather than mere paraphrase variation; if that proxy fails, the entropy estimate stops tracking hallucination.","fun_headline_variants_meta":{"raw":{"variants":["Pairwise similarity beats semantic entropy for LLM uncertainty","Log-sum-exp over answer similarities flags LLM hallucinations","SNNE: a pairwise uncertainty measure that generalizes semantic entropy","From cluster centroids to pairwise similarities: better LLM uncertainty","One simple formula replaces cluster entropy for LLM uncertainty"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000498,"raw_usage":{"total_tokens":2435,"prompt_tokens":938,"completion_tokens":1497,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":1417}},"tokens_in":554,"tokens_out":1497,"duration_ms":10779,"temperature":1.0,"reasoning_tokens":1417,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:08:43.859277+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a set of long one-sentence questions whose correct answers are phrased entirely differently from the model's sampled responses (high semantic equivalence, near-zero ROUGE-L); if SNNE then assigns high uncertainty to those correct answers, or its AUROC for hallucination detection falls below that of semantic entropy, the central claim fails.","supporting_citations":[],"review_version":1}