{"id":"764d8513-6dd6-43d6-801d-cc4a5bdde12b","arxiv_id":"2509.03533","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"UDIB, an upper-bounded Deterministic Information Bottleneck clustering algorithm, is applied to LLM prompt and response embeddings to produce topic spaces that sharpen semantic divergence metrics for hallucination analysis.","lead":"This paper introduces UDIB, an entropy-regularized clustering method that approximates Deterministic Information Bottleneck clustering for high-dimensional embeddings, and uses it to build shared topics for prompts and responses in LLM conversations. The authors claim the resulting topics make semantic divergence metrics more sensitive for detecting confabulations, though the experimental comparison is partly confounded by different embedding settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline SDM comparison confounds clustering method with embedding model; UDIB's reported sensitivity gains may be an embedding artifact.","rationale":"The paper's central claim is that UDIB provides a 'superior foundation' for SDM and a 'more sensitive tool' for confabulation detection. The only empirical support is a side-by-side comparison of SDM scores computed with UDIB against scores 'reproduced' from the author's earlier paper [3]. The new experiments explicitly use Qwen3-Embedding-0.6B, but the baseline's embedding model is unspecified. If the baseline used different embeddings, the comparison conflates the clustering algorithm with the embedding representation. This is the single most load-bearing concern because the entire novelty rests on showing UDIB beats agglomerative clustering for SDM; without controlling for the embedding, no such conclusion follows. The topic-count mismatch adds a second, independent confound, but the embedding issue alone is sufficient. The reader's weakest_assumption is phrased identically, so I agree with it. A concrete rerun with matched embeddings and k values would settle the issue. This does not require a verdict change—the paper is already CONDITIONAL—so I set verdict_should_be to UNCHANGED while emphasizing the condition.","tokens_in":18385,"tokens_out":3651,"duration_ms":38251,"concrete_test":"Using the same Qwen3-Embedding-0.6B embeddings and the SDM code from [3], rerun agglomerative clustering on the Set A and Set B prompts with topic counts matched to the UDIB runs (Set A: k=8,9,9; Set B: k=7,7,5,8). Recompute the SH, Φ, and KL metrics reported in Tables 3 and 6, ideally over multiple seeds with error bars. If the monotonic Set B SH trend (0.1628→0.2315→0.2924) and the dynamic-range improvements disappear or shrink substantially, the UDIB advantage is an artifact of the embedding model or of k mismatch, not of the clustering algorithm.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that UDIB topic spaces make SDM metrics more sensitive and discriminative—rests on a comparison between UDIB-computed SDM scores (Tables 3 and 6) and baseline scores 'reproduced' from [3] (Tables 2 and 5). Section 5 states that 'sentence embeddings are done using the Qwen3-Embedding-0.6B model' for this paper, but it never states the embedding model used in [3]. If the baseline used a different embedding model, then differences in cluster geometry, topic assignments, and all entropy-based SDM metrics (H(P), Φ, JSD, KL, SH) could be caused by the embedding change alone, not by UDIB's clustering algorithm. This is compounded by a second confound: the UDIB tables use different topic counts (Set A: k=8,9,9; Set B: k=7,7,5,8) than the baseline, whose k values are not even reported. SDM scores depend heavily on k because they are computed over the topic distribution. The paper's Sec. 5.4 claims UDIB 'enhances sensitivity and dynamic range' based on this uncontrolled comparison. If the baseline were rerun on the same Qwen3 embeddings with matched k, the monotonic Set B trend and the larger dynamic range might disappear, invalidating the paper's key empirical claim. The reader's weakest_assumption identifies exactly this issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UDIB, a tractable version of Geometric Deterministic Information Bottleneck clustering, obtained by replacing the KL divergence between a Gaussian and a Gaussian mixture with the Hershey-Olsen convexity bound. This yields an entropy-regularized K-means-like assignment rule with a single effective temperature tau = 2s^2/beta. The method is applied to joint clustering of Qwen3-Embedding sentence embeddings of prompts and responses, with the number of topics selected by a 'kink-angle' heuristic over multiple seeds. SDM scores computed on UDIB topics are compared with scores from an earlier agglomerative-clustering baseline; the paper claims that the UDIB topic space is more interpretable and makes SDM metrics more sensitive and discriminative, especially through monotonic increases across Set B task complexity.","tokens_in":18736,"tokens_out":7026,"duration_ms":77502,"significance":"The mathematical core is mostly sound and presented with useful detail: the upper-bound substitution is correctly derived (Eqs. 10-14), and the Jensen comparison with K-means (Eq. 17) is correct. The method genuinely reduces the DIB optimization to a practical algorithm with one effective hyperparameter and gives a reproducible initialization/selection protocol (Algorithm 1, multi-seed procedure). If the empirical claims were supported by a controlled comparison, this would be a useful advance for LLM faithfulness-hallucination detection. However, the current experimental evidence does not support the headline sensitivity claims because the baseline is confounded and the reported SDM values lack uncertainty quantification.","major_comments":[{"comment":"The central empirical claim of enhanced sensitivity rests on comparisons to Tables 2 and 5, reproduced from [3]. Section 5 specifies Qwen3-Embedding-0.6B for all experiments in this paper, but the embedding model (and k) used in [3] is never given. If the baseline used different embeddings, every SDM entry changes even with identical clustering. Moreover, the UDIB tables use different topic counts (Set A: k=8,9,9; Set B: 7,7,5,8) than the baseline, whose k is not reported; H(P), JSD and KL are all functions of k. The Section 5.4 statements about 'wider dynamic range' and the monotonic Set B trend are therefore not attributable to UDIB without rerunning AC on Qwen3 embeddings with matched k and reporting the baseline k.","section":"Section 5.2.1, Section 5.4, Tables 2/3 and 5/6"},{"comment":"The identity I(c;x)=H[c] requires H[c|x]=0. In the paper's model p(x|i)=N(x; x_i, s^2 I) with finite s, x does not determine the index i uniquely, so p(c|x) is a mixture over indices and H[c|x]>0 in general. The equality holds only in the s->0 limit, which is not the regime used by the algorithm (tau=2s^2/beta is finite). Since Eq. (18) is the stated justification for the information profile used by the kink-angle selector, the model-selection diagnostic needs a corrected derivation or an explicit estimate of H[c|x].","section":"Section 4.5, Eq. (18)"},{"comment":"The UDIB SDM tables report single numbers without standard errors or confidence intervals, despite the multi-seed design. Table 1 shows large seed-to-seed dispersion (e.g., Rec. # Clusters 10.00 +/- 1.84 and Kink angle 33.70 +/- 22.56 for High Stability). Since the final k and assignments vary across seeds, all SDM metrics inherit this variability; the claimed monotonic increases in Set B (SH 0.1628, 0.2315, 0.2924) could lie within the run-to-run noise. Report per-seed SDM distributions or a sensitivity analysis over the M runs.","section":"Tables 3 and 6; Section 5.2"},{"comment":"There is a discrepancy in the number of topics used for SDM. Table 3 uses k=9 for both Moderate and Low Stability, while Appendix A.2 and A.3 report optimal nc=10 and nc=11, and Table 1 gives Kink Angle mean recommendations of 10.5 and 8.6. The mode-based rule in Section 5.5 is not documented with the actual per-run nc histogram, so the reader cannot see how the final k values were obtained. This is not merely cosmetic: every SDM value in Tables 3 and 6 depends on k.","section":"Section 5.5 vs. Table 3 and Appendix A"}],"minor_comments":[{"comment":"Typographical issues include 'promts' (Introduction), 'We For all experiments' (Section 5.2.1), 'CSummary' (Table 1 header), 'F actual' (Table 5), and 'V ariability' (Section 5.5).","section":"Throughout"},{"comment":"The heatmaps would be more informative with a shared color scale and axis labels for topic indices. The qualitative descriptions are somewhat subjective and hard to verify from the small figures.","section":"Figures 2 and 4"},{"comment":"The notation E_{i,c,j in S_c} is informal; clarify the dependence on cluster c and which quantities are random. Also, the phrase 'total loss function' should be defined consistently with the per-point Lagrangian in Eq. (14).","section":"Equation (15)"}],"recommendation":"major_revision","confidential_remarks":"The experimental section relies entirely on a comparison with the author's own earlier work [3], and the baseline conditions are not controlled. Before resubmission, the author should rerun the baseline under matched embedding and k settings. Also, references [6] and [10] have arXiv identifiers that look implausibly round; the editor may wish to verify that these citations are real and accurately described."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe takeaway: the UDIB algorithm is real and the math checks out; the empirical case that it makes SDM more sensitive is not established because the baseline comparison is uncontrolled.\n\nWhat's new: substituting the Hershey-Olsen convexity bound for the intractable KL between a Gaussian and a GMM in the DIB assignment step is a neat trick. It turns DIB into a practical iteration that is essentially K-means with a pairwise-distance term plus an entropy penalty, and it collapses s and beta into one temperature-like parameter. The Jensen inequality argument linking the loss to K-means is correct. The multi-seed procedure and the kink-angle model selection are reasonable engineering. The qualitative topic analysis in the appendix is actually the most convincing part: the UDIB clusters separate semantic content from prompt artifacts (CONSTRAINTS markers, JSON) and split Hamlet's feigned vs. Ophelia's genuine madness. That suggests the algorithm is doing something right.\n\nWhere it gets soft: the headline claim of superior confabulation sensitivity rests on comparing UDIB SDM scores (Tables 3, 6) against numbers 'reproduced' from the author's earlier paper [3]. The paper never says what embedding model [3] used. This paper uses Qwen3-Embedding-0.6B. If the baseline used a different embedding model, the comparison is embedding-vs-embedding, not clustering-vs-clustering. On top of that, the number of topics k is not matched: Table 3 uses k=8,9,9 while Table 1's recommended k for the same prompts under the kink heuristic is 10, 10-11, 9-ish; for Set B the k values differ across prompts (7,7,5,8) and the baseline k isn't reported. Since entropy-based SDM metrics depend on k, the 'monotonic trend' and 'dynamic range' claims could be an artifact. Also, the SDM tables have no error bars, and the Wasserstein rows are numerically identical between baseline and UDIB, which makes me wonder whether those were computed on raw embeddings (legit) or copied (sloppy).\n\nOne point where I disagree with the reader's report: Eq (18), H[c|x]=0, is fine for a deterministic assignment rule—given x, c is a function of x. That's not a flaw.\n\nWho should read this: people working on information-bottleneck clustering and anyone building faithfulness detectors who wants a principled alternative to vanilla K-means. The algorithmic contribution is worth citing. The experimental claim needs a redo: same embeddings for baseline, matched k, error bars, and code.\n\nRecommendation: send to peer review. The algorithm is sound and the application is relevant; a good referee would ask for the same-embedding baseline and uncertainty quantification. That's a fixable revision, not a desk reject.","headline":"UDIB is a sound, practical twist on DIB clustering, but the claimed SDM sensitivity gains rest on a confounded baseline comparison.","tokens_in":19194,"tokens_out":2743,"would_cite":true,"duration_ms":28629,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50","62H30","94A17"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that topic identification for LLM hallucination detection should be reformulated as a deterministic information bottleneck problem, and delivers a tractable algorithm, UDIB, that sharpens Semantic Divergence Metrics by mak","keywords":["deterministic information bottleneck","UDIB","topic identification","semantic divergence metrics","confabulation detection","LLM hallucinations","embedding clustering","information bottleneck"],"falsifier":"Fix one embedding model and the same SDM pipeline; run agglomerative clustering and UDIB on the same Set A and Set B prompt–response data with comparable model-selection rules. If the monotonic Set B gradient (0.1628 < 0.2315 < 0.2924) and the wider dynamic range disappear when embeddings are held fixed, UDIB's claimed sensitivity gain dissolves.","tokens_in":18283,"feed_emoji":"🧠","tokens_out":5840,"duration_ms":56912,"temperature":0.7,"pith_summary":"The paper claims that the standard way to find topics in LLM prompt–response pairs—geometric clustering of sentence embeddings—is mismatched to the downstream information-theoretic analysis, and fixes the mismatch by deriving a practical Deterministic Information Bottleneck algorithm. The algorithm, UDIB, is an entropy-regularized K-means that trades spatial tightness against cluster parsimony and is directly optimized to keep the topic representation informative about the prompt–response relationship. Applied within the Semantic Divergence Metrics framework, it yields cleaner topics, more interpretable co-occurrence maps, and scores that separate task types more sharply—for closed-domain prompts the divergence measures become monotonic in task complexity, which the older clustering did not show. If correct, this gives confabulation detection a more sensitive measurement tool without overturning the SDM framework's conclusions.","feed_headline":"One clustering tweak makes LLM hallucination scores more sensitive","feed_subtitle":"Entropy-regularized K-means topics reveal task-complexity gradients that agglomerative clustering misses.","key_machinery":"UDIB (Upper-bounded Deterministic Information Bottleneck): an iterative hard-clustering algorithm whose per-point loss is (1/n_c) sum_{j in c} ||x_i - x_j||² − τ log q(c), with τ = 2s²/β. It replaces DIB's intractable KL term with a Jensen/Hershey–Olsen upper bound, collapsing smoothing scale s and tradeoff β into one temperature-like parameter; the number of clusters is selected by the Kink Angle heuristic on the information profile H[c].","core_discovery":"The paper's central claim is that the topics used to compare LLM prompts with their responses should be chosen by how much information they carry about the prompt–response relationship, not by spatial proximity alone. It makes the Deterministic Information Bottleneck practical for high-dimensional embeddings by replacing the intractable KL divergence between a Gaussian and a Gaussian mixture with the Hershey–Olsen convexity upper bound. The resulting UDIB assignment rule—minimize the mean pairwise squared distance within a cluster plus τ times the negative log cluster size—is an entropy-regularized K-means that prunes uninformative clusters on its own. Fed into the Semantic Divergence Metric","pith_inferences":["Swapping the clustering algorithm is not the only change between the baseline and new tables; the sentence embeddings are also different. A clean test would fix the embedding model and vary only the clustering method to confirm that UDIB itself, not the new embeddings, drives the sharper SDM scores.","The Kink Angle heuristic's recommended cluster count varies widely across seeds (e.g., 10.50 ± 1.50), so the final k is a meta-statistic over runs. A natural extension would quantify how SDM scores fluctuate when k is perturbed within the stability region of τ.","The single effective temperature τ = 2s²/β means the smoothing scale and the information–compression tradeoff are not separately identifiable; probing with several τ values would show whether topic granularity, rather than the entropy penalty, is doing the work.","The information profile diagnostics could be repurposed as a test-time signal: monitoring the kink angle across dialogue turns might detect topic drift or the onset of confabulation without ground-truth labels."],"forward_implications":["If UDIB topics are used, SDM remains coherent: the core ordinal conclusions—high vs moderate vs low stability in Set A, and forced hallucination as a low-scoring outlier in Set B—are preserved.","The widened dynamic range means tasks can be graded continuously: factual recall, comparison, and forecasting now occupy ordered values on the divergence scale instead of overlapping.","A low-and-stable score on a nonsensical prompt becomes an explicit 'confident confabulation' signature, distinguishable from semantically unstable exploration.","Built-in model selection removes the need to fix the number of topics beforehand, making topic identification more parsimonious.","Because UDIB is a modified K-means with one effective hyperparameter, it can be dropped into existing embedding-based pipelines beyond LLM input-output analysis."],"supporting_citations":[{"why":"Introduces the Deterministic Information Bottleneck objective and its hard-clustering preference for few, informative clusters that UDIB descends from.","marker":"[7]"},{"why":"Shows how DIB specializes to geometric clustering and to K-means-like algorithms, and supplies the iterative assignment equations that UDIB modifies.","marker":"[8]"},{"why":"Supplies the convexity upper bound on KL divergence between Gaussian mixtures that turns DIB's intractable term into a closed-form, high-dimensional tractable loss.","marker":"[4]"},{"why":"Gives the SDM framework, the Set A and Set B prompt sets, and the baseline agglomerative-clustering scores that UDIB results are directly compared against.","marker":"[3]"},{"why":"Provides the Semantic Entropy baseline that motivates divergence-style hallucination detection and that SDM extends.","marker":"[2]"},{"why":"Defines the information bottleneck objective that the whole method instantiates.","marker":"[9]"}],"fun_headline_variants":["Entropy-regularized K-means boosts confabulation detection","Better topics from information bottleneck sharpen LLM hallucination checks","Smarter clustering reveals hidden LLM confabulations","Information-theoretic clustering improves LLM truthfulness checks"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The headline comparisons reuse SDM numbers from an earlier study that clustered with a different algorithm and, in all likelihood, different sentence embeddings; if the embeddings differ, UDIB's apparent advantage could be an artifact of the embeddings rather than of the clustering method, so the paper's central claim would not be established.","fun_headline_variants_meta":{"raw":{"variants":["Entropy-regularized K-means boosts confabulation detection","Better topics from information bottleneck sharpen LLM hallucination checks","Smarter clustering reveals hidden LLM confabulations","Information-theoretic clustering improves LLM truthfulness checks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000607,"raw_usage":{"total_tokens":2682,"prompt_tokens":774,"completion_tokens":1908,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":1850}},"tokens_in":518,"tokens_out":1908,"duration_ms":13862,"temperature":1.0,"reasoning_tokens":1850,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:47:23.429866+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix one embedding model and the same SDM pipeline; run agglomerative clustering and UDIB on the same Set A and Set B prompt–response data with comparable model-selection rules. If the monotonic Set B gradient (0.1628 < 0.2315 < 0.2924) and the wider dynamic range disappear when embeddings are held fixed, UDIB's claimed sensitivity gain dissolves.","supporting_citations":[{"cited_title":"Strouse and D","cited_arxiv_id":null,"evidence_quote":"Introduces the Deterministic Information Bottleneck objective and its hard-clustering preference for few, informative clusters that UDIB descends from."},{"cited_title":"The information bottleneck and geometric clustering","cited_arxiv_id":"1712.09657","evidence_quote":"Shows how DIB specializes to geometric clustering and to K-means-like algorithms, and supplies the iterative assignment equations that UDIB modifies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the convexity upper bound on KL divergence between Gaussian mixtures that turns DIB's intractable term into a closed-form, high-dimensional tractable loss."},{"cited_title":"Halperin","cited_arxiv_id":null,"evidence_quote":"Gives the SDM framework, the Set A and Set B prompt sets, and the baseline agglomerative-clustering scores that UDIB results are directly compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Semantic Entropy baseline that motivates divergence-style hallucination detection and that SDM extends."},{"cited_title":"Tishby, F","cited_arxiv_id":null,"evidence_quote":"Defines the information bottleneck objective that the whole method instantiates."}],"review_version":1}