Pith. sign in

REVIEW 4 major objections 4 minor 77 references

A Bayesian model that clusters prompts by semantic similarity corrects prompt dependence in LLM security benchmarks, giving trustworthy uncertainty estimates for model comparison.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:16 UTC pith:74PTVZBM

load-bearing objection A plausible Bayesian model for LLM security evals with unknown prompt clusters, but the central claim that it corrects prompt dependence is not actually validated — worth peer review, not acceptance as-is. the 4 major comments →

arxiv 2510.05709 v2 pith:74PTVZBM submitted 2025-10-07 cs.CR cs.AIcs.CL

Correcting Prompt Dependence in LLM Benchmarks: A Bayesian Hierarchical Model with Embedding-Space Clustering

Mary Llewellyn , Isobel Thornton , James Bishop , Annie Gray This is my paper
classification cs.CR cs.AIcs.CL
keywords Bayesian hierarchical modelembedding-space clusteringprompt dependenceuncertainty quantificationLLM security evaluationprompt injectionadversarial robustnesslimited-data inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that standard LLM evaluation metrics mislead when test prompts are interdependent and the number of evaluations is small, two conditions common in adversarial-robustness testing. To fix this, it proposes a Bayesian hierarchical model that first clusters prompts in embedding space, treating the number of clusters as unknown, and then estimates a separate attack-success probability for each cluster. The authors validate the model by showing it predicts held-out outcomes better than the same model without clustering, and they demonstrate the full pipeline by comparing Transformer and state-space architectures under prompt-injection attacks. The practical payoff is that practitioners can draw conclusions about model vulnerability with honest uncertainty intervals instead of overconfident point estimates.

Core claim

The central claim is that the reliability of LLM security evaluations can be improved by explicitly modelling the semantic structure of test prompts. The model embeds each prompt, partitions the prompts into an unknown number of clusters via agglomerative clustering with Spearman similarity, and assumes that within a cluster all prompts share the same probability of attack success. The overall attack-success rate is the average of the cluster-level probabilities, and the posterior is sampled by importance sampling. In 5-fold cross-validation on four prompt-injection attacks, this model produced higher expected log predictive densities than the identical model with no clustering, for both a T

What carries the argument

The load-bearing mechanism is the combination of embedding-space clustering with a Bayesian hierarchical Beta-Binomial model. Prompts are mapped to sentence embeddings and clustered using agglomerative clustering with Spearman correlation; the number of clusters is treated as unknown with a diffuse prior, and cluster assignments are treated as deterministic given the cluster count. A Beta(1,1) prior on each cluster's attack-success probability and a Beta-Binomial marginal likelihood make the posterior tractable via importance sampling, yielding weighted samples of the cluster probabilities, the cluster count, and the average attack-success rate. This lets the model pool information across si

Load-bearing premise

The method's correction for prompt dependence rests on the assumption that clustering by embedding similarity partitions prompts into statistically independent topics, so that all prompts in a cluster have the same attack-success probability; if that assumption fails, the model's improved fit may just come from added flexibility.

What would settle it

Take a benchmark whose prompts have known ground-truth topics and, for each cluster produced by the model, check whether the within-cluster attack-success rates are actually homogeneous and whether the recovered clusters align with the known topics. If prompts that the model places in one cluster show substantially different success probabilities, or if the posterior similarity matrix fails to separate known-distinct topics, the claimed correction for prompt dependence is not the source of the model's predictive improvement.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Practitioners can obtain more trustworthy uncertainty intervals for attack-success rates when they only have tens of repeated queries per prompt.
  • Benchmark results no longer require the number of prompt topics to be known in advance, making the analysis scalable to many attacks.
  • The comparison of architectures becomes more reliable: findings like 'Transformer more vulnerable to divergence attacks' are stated with credible intervals that reflect prompt interdependence.
  • The same model can be applied to any prompt-based evaluation task, including accuracy benchmarks, by clustering across tasks as demonstrated in the math-performance case study.
  • Cross-validation indicates the model's predictive gains are largest when prompts are highly interdependent, suggesting the method directly targets the bias it was designed to remove.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same embedding-space clustering could be used as a diagnostic before running an evaluation, to identify redundant prompts and design leaner benchmarks.
  • If the independence-within-cluster assumption holds, the model could be extended to non-binary outcomes (e.g., partial success or graded responses) by replacing the Binomial likelihood with a categorical or ordinal likelihood.
  • The approach could combine with automated judges to handle attacks whose success cannot be determined by string matching, since clustering does not depend on the validation method.
  • One testable extension is to replace the prior on the number of clusters with a Dirichlet-process prior, though the paper's fixed-prior importance sampler already avoids that complexity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a Bayesian hierarchical model for LLM security evaluations in limited-data settings, with embedding-space clustering of prompts to address prompt dependence. The number of clusters S is treated as unknown with a Binomial-derived prior; agglomerative clustering on sentence embeddings partitions prompts, and a within-cluster attack-success probability is assumed constant. The overall attack success probability pa is defined as the equal-weight average of cluster probabilities, and posterior inference is performed by an importance-sampling scheme with beta-binomial marginal-likelihood weights. Validation is carried out by 5-fold cross-validation comparing expected log predictive densities (ELPD) against a no-clustering (S=1) baseline on four Garak attacks for two 2.8b models. The paper then applies the pipeline to compare Mamba, Transformer, and hybrid models matched by training data or by mathematical performance.

Significance. If the central claim were established, the model would be a useful contribution to LLM security evaluation: it addresses real problems of limited repetition, non-deterministic outputs, and inter-dependent test prompts. The strengths of the manuscript are the correct importance-sampling derivation in Appendix B, the practical framing of experimental design (training-matched and performance-matched comparisons), and the qualitative PSM evidence that embedding-space clusters are semantically interpretable. The case study also demonstrates a sensible workflow for comparing architectures with posterior credible intervals. However, the paper's headline claim of 'correcting prompt dependence' is not validated against ground truth, and the abstract contains quantitative MAE claims that do not appear in the body. The core modelling idea is plausible, and the deficiencies are addressable, but the present manuscript overstates what the evidence supports.

major comments (4)
  1. [Section V-D / Table II] The ELPD comparison against the S=1 model does not establish that prompt dependence has been corrected. In the absence of ground-truth attack success rates (as the paper itself acknowledges in V-D), a model with more clusters and more parameters will generally improve predictive density even if the clusters are arbitrary partitions of the embedding space. The reported ELPD gains are therefore consistent with added flexibility rather than with a genuine correction for prompt dependence. The paper should either add a synthetic-data validation with known cluster-specific success probabilities, or substantially soften the claim to 'improved out-of-sample predictive performance under the proposed clustering model'.
  2. [Abstract vs. full text] The abstract claims '4-73% improvements to mean absolute errors and 40-450 unit improvements to expected log posterior densities', but the full text reports only expected log posterior densities (Table II). No mean absolute error results appear anywhere in the paper. The MAE improvement figures are therefore unsupported as submitted. The abstract must be brought into line with the actual reported experiments, or the missing MAE experiments must be added and reported.
  3. [Section IV-B, Eq. (pa = (1/S) Σ p_k)] The central correction is partly definitional. The target pa is defined as the equal-weight average of cluster-wise success probabilities, and the clusters themselves are produced by the same embedding-space representation that is used to claim 'dependence'. The assumption that prompts within a cluster have equal attack-success probability is stipulated ('We stipulate that...'), not tested. If the clusters do not correspond to statistically independent topic groups, the equal-weighted pa can distort the overall success rate, and the ELPD gain may simply reflect the added flexibility. The model should be validated in a setting where the true per-prompt success probabilities and cluster structure are known, e.g., a simulation study, so the 'correction' claim can be separated from the flexibility effect.
  4. [Section V-D, cross-validation procedure] The 5-fold cross-validation uses 'coarse labelling' to stratify folds, with the statement that this 'does not inform or bias the proposed model directly here'. This is not demonstrated. If the coarse labels correlate with the embedding-space clusters, the stratification can make the predictive task easier for the clustering model in a way that is unrelated to its real-world use, where such labels are assumed unavailable. The authors should report results with unstratified random folds or otherwise show that the ELPD comparison is invariant to the stratification scheme.
minor comments (4)
  1. [Algorithm 1] In the line 'set S_t = min(n, B + 1)', the variable B should be B_t, the sampled value, for clarity.
  2. [Section IV-B] The notation x_i is used both for the binomial random variable (x_i ~ Binomial(m_i, p_k)) and for the observed count. Please distinguish observed counts from random variables, e.g., with a lowercase/uppercase convention.
  3. [Title] The arXiv title ('Correcting Prompt Dependence in LLM Benchmarks...') differs from the full-text title ('Towards Reliable and Practical LLM Security Evaluations via Bayesian Modelling'). The titles should be aligned, especially given that the arXiv title makes a stronger claim than the body currently supports.
  4. [Section V-D] The paragraph describing 'approximate stratification' is vague about how many coarse classes were used and how folds were constructed. A brief algorithmic description would help reproducibility.

Circularity Check

2 steps flagged

Partial circularity: cluster-independence validation is tautological and the corrected pa is defined through the same embeddings used to define dependence.

specific steps
  1. self definitional [Section IV-B, definition of pa after the cluster model]
    "We stipulate that the prompts within each cluster now have close semantic ties and thus model the probability of attack success as the same within each cluster. ... pa = 1/S \sum_{k=1}^S pk."

    The claimed correction for prompt dependence is implemented by defining the aggregate attack probability pa as the equal-weighted average of cluster-wise probabilities pk, where the clusters are constructed from the same embedding-space semantic similarity that is treated as the source of prompt dependence. Thus, by construction, within-cluster dependence is removed by equating pk for all prompts in a cluster, and the 'correction' is stipulated rather than derived from data. The paper itself acknowledges in Section V-D that there is no ground-truth attack success rate with which to validate the cluster structure.

  2. other [Section V-D, Figure 2 and PSM validation]
    "Importantly, Figure 2 shows clear between-cluster independence. The identification and independence of clusters validates the need to cluster prompt embeddings to trust quantification of uncertainty in these attacks."

    The posterior similarity matrix is computed from the model's own posterior cluster assignments, which by construction partition prompts into disjoint clusters (each index belongs to exactly one cluster). Consequently, any pair of prompts in different clusters has co-clustering probability exactly zero, so 'clear between-cluster independence' is guaranteed by the partition definition rather than demonstrated empirically. Using this tautology to validate the independence assumption is circular: the clusters are shown to be independent only because the model forces them to be disjoint.

full rationale

The paper does not rely on a self-citation chain, and the main predictive validation is an out-of-sample ELPD comparison against the S=1 model, which is not a fitted-to-target circularity. However, two steps undermine the 'correction for prompt dependence' claim. First, the aggregate pa is defined as the average of cluster-wise probabilities, where the clusters are derived from the same embedding-space similarity that is used to characterize prompt dependence; the correction is therefore built into the definition rather than independently established. Second, the validation of cluster independence via the posterior similarity matrix is tautological, because the model's cluster assignments are disjoint by construction, making between-cluster co-clustering zero by definition. These are partial circularities: the predictive ELPD improvements retain independent empirical content, but the claim that the model corrects for prompt dependence is not anchored to an external benchmark or ground-truth cluster structure. Score 4 reflects this partial, non-central circularity.

Axiom & Free-Parameter Ledger

1 free parameters · 6 axioms · 1 invented entities

The central model relies on several hand-chosen assumptions: the Binomial prior on the number of clusters S, the Beta(1,1) cluster-probability priors, and the identification of semantic embedding clusters with statistically independent topics. The last is the most consequential: it is load-bearing for the 'correcting prompt dependence' claim, yet is not validated against ground-truth independent prompt sets. No code or data is provided to check the implementation.

free parameters (1)
  • S prior hyperparameters: B ~ Binomial(50n, 0.01), S = min(n, B+1) = Prior mean of B is 0.5n; standard deviation about sqrt(0.495n)
    The 50 and 0.01 constants are chosen by hand to make a 'diffuse' prior over the number of clusters. They are not derived from data or theory, and they strongly center S near n/2, which may dominate the posterior when n is small.
axioms (6)
  • domain assumption Within each cluster C_k, all prompts share the same attack-success probability p_k.
    Section IV-B: 'We stipulate that the prompts within each cluster now have close semantic ties and thus model the probability of attack success as the same within each cluster.' This exchangeability assumption is load-bearing: if prompts in a cluster have heterogeneous success rates, the cluster-level binomial is misspecified.
  • domain assumption Semantic embedding similarity (all-MiniLM-L6-v2 + Spearman correlation + agglomerative clustering) partitions prompts into independent topics.
    Sections IV-A and IV-B use embeddings and clustering to define prompt independence. The paper provides visual PSM evidence but no ground-truth validation that clusters correspond to statistically independent prompt topics.
  • ad hoc to paper The Binomial(50n, 0.01) prior on S is adequately diffuse and does not dominate the posterior.
    Section IV-B introduces this prior without derivation or sensitivity analysis. The choice of 50n and 0.01 is arbitrary and may influence the inferred number of clusters.
  • domain assumption Cluster assignments can be treated as deterministic given S.
    Section IV-B: 'we do not treat cluster assignments as random variables given the number of clusters, S, as this leads to over-parameterisation with unstable results.' This ignores clustering uncertainty, which could understate posterior uncertainty in pa.
  • domain assumption The overall attack success rate is the unweighted mean of cluster probabilities: pa = (1/S) Σ pk.
    Section IV-B defines pa this way. It gives equal weight to each inferred topic, regardless of the number of prompts in the cluster or the precision of pk.
  • ad hoc to paper Coarse labelling used for cross-validation stratification does not bias the model comparison.
    Section V-D uses human coarse labels to build stratified folds. The authors assert this does not bias the model, but if the labels correlate with the embedding clusters, the clustered model can be artificially advantaged in the ELPD comparison.
invented entities (1)
  • S, the unknown number of semantic topics/clusters no independent evidence
    purpose: Latent variable that avoids assuming a known number of prompt topics, allowing the model to pool repeated trials within inferred clusters.
    S is a model construct with an ad hoc Binomial prior; its posterior is estimated from data, but there is no external benchmark showing that S recovers true independent prompt topics.

pith-pipeline@v1.3.0-alltime-deepseek · 21081 in / 11020 out tokens · 94679 ms · 2026-08-04T11:16:18.097242+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Correcting Prompt Dependence in LLM Benchmarks: A Bayesian Hierarchical Model with Embedding-Space Clustering." pith.science (2026). https://pith.science/paper/74PTVZBM

@misc{pith2026251005709,
  author       = {Pith},
  title        = {Pith review of: Correcting Prompt Dependence in LLM Benchmarks: A Bayesian Hierarchical Model with Embedding-Space Clustering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/74PTVZBM}},
  note         = {Machine review of arXiv:2510.05709}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

LLM benchmarking metrics often misstate performance and uncertainty as they rely on two assumptions that frequently do not hold in practice: (i) a sufficient number of evaluations are available for classical inference, and (ii) test prompts are independent. We propose a corrective Bayesian hierarchical model with embedding-space clustering that provides robust performance metrics in limited-data settings while correcting for prompt dependence. We apply the approach to adversarial robustness benchmarks, showing consistent recovery of clustering structure, resulting in more reliable performance metrics, with 4-73% improvements to mean absolute errors and 40-450 unit improvements to expected log posterior densities.

Figures

Figures reproduced from arXiv: 2510.05709 by Annie Gray, Isobel Thornton, James Bishop, Mary Llewellyn.

Figure 1
Figure 1. Figure 1: Visualisation of the prompts from the Package hallucination (JavaScript) attack [12]. From left to right: (1) embedding when PCA is applied, (2) embedding when t-SNE is applied, and (3) shows the percentage of variance explained by each PC [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Cluster label PSM for the Package hallucination (JavaScript) attack and 2.8b parameter Mamba model. The labels on the x axis are in the same order as those on the y axis. For presentation, the presented label orders are found via hierarchical clustering with average linkage, but this does not change the data presented. We use 5-fold cross-validation with approximate stratifica￾tion to demonstrate out-of-sa… view at source ↗
Figure 3
Figure 3. Figure 3: Results for each attack, showing average posterior means and [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Results for each attack versus accuracy, showing average posterior means and 90% credible intervals. The legend can [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Visualisation of the prompts from each attack. From left to right: (1) embedding when PCA is applied, (2) embedding [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Cluster label PSMs for each attack and the 2.8b parameter Mamba model. Labels on the x axis appear in the same [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Cluster label PSMs for each attack and the 2.8b parameter Transformer. Labels on the x axis appear in the same order as [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 33 linked inside Pith

  1. [1]

    Codestral Mamba, July 2024

    Mistral AI. Codestral Mamba, July 2024. https://mistral.ai/news/ codestral-mamba

    2024

  2. [2]

    Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks, June

    Melissa Ailem, Katerina Marazopoulou, et al. Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks, June

  3. [3]

    Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems.Annals of Statistics, 2(6), November 1974

    Charles Antoniak. Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems.Annals of Statistics, 2(6), November 1974

    1974

  4. [4]

    xLSTM: Extended long short- term memory, December 2024

    Maximilian Beck, Korbinian P ¨oppel, et al. xLSTM: Extended long short- term memory, December 2024. arXiv:2405.04517

    Pith/arXiv arXiv 2024

  5. [5]

    D. A. Binder. Bayesian cluster analysis.Biometrika, 65(1), April 1978

    1978

  6. [6]

    Blackwell, Jon Barry, and Anthony G

    Robert E. Blackwell, Jon Barry, and Anthony G. Cohn. Towards repro- ducible LLM evaluation: Quantifying uncertainty in LLM benchmark scores, June 2025. arXiv:2410.03492

    Pith/arXiv arXiv 2025

  7. [7]

    Sam Bowyer, Laurence Aitchison, and Desi R. Ivanova. Position: Don’t use the CLT in LLM evals with fewer than a few hundred datapoints, May 2025. arXiv:2503.01747

    Pith/arXiv arXiv 2025

  8. [8]

    On evaluating adversarial robustness, February 2019

    Nicholas Carlini, Anish Athalye, et al. On evaluating adversarial robustness, February 2019. arXiv:1902.06705

    Pith/arXiv arXiv 2019

  9. [9]

    A survey on evaluation of large language models, December 2023

    Yupeng Chang, Xu Wang, et al. A survey on evaluation of large language models, December 2023. arXiv:2307.03109

    Pith/arXiv arXiv 2023

  10. [10]

    Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality, May 2024

    Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality, May 2024. arXiv:2405.21060

    Pith/arXiv arXiv 2024

  11. [11]

    A cluster separation mea- sure.IEEE transactions on pattern analysis and machine intelligence, (2):224–227, 2009

    David L Davies and Donald W Bouldin. A cluster separation mea- sure.IEEE transactions on pattern analysis and machine intelligence, (2):224–227, 2009

    2009

  12. [12]

    garak: A framework for security probing large language models, June 2024

    Leon Derczynski, Erick Galinkin, et al. garak: A framework for security probing large language models, June 2024. https://github.com/leondz/ garak

    2024

  13. [13]

    LLM-evaluation tropes: Perspectives on the validity of LLM-evaluations, April 2025

    Laura Dietz, Oleg Zendel, et al. LLM-evaluation tropes: Perspectives on the validity of LLM-evaluations, April 2025. arXiv:2504.19076

    arXiv 2025

  14. [14]

    Understanding robustness of visual state space sodels for image classification, March 2024

    Chengbin Du, Yanxi Li, and Chang Xu. Understanding robustness of visual state space sodels for image classification, March 2024. arXiv:2403.10935

    Pith/arXiv arXiv 2024

  15. [15]

    Skewed score: A statistical framework to assess autograders, July 2025

    Magda Dubois, Harry Coppock, et al. Skewed score: A statistical framework to assess autograders, July 2025. arXiv:2507.03772

    arXiv 2025

  16. [16]

    Aparna Elangovan, Ling Liu, et al. ConSiDERS – the human-evaluation framework: Rethinking human evaluation for generative large language models.Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 1:1137–1160, August 2024

    2024

  17. [17]

    Pythia-2.8b, April 2023

    EleutherAI. Pythia-2.8b, April 2023. https://huggingface.co/EleutherAI/ pythia-2.8b

    2023

  18. [18]

    A density-based algorithm for discovering clusters in large spatial databases with noise

    Martin Ester, Hans-Peter Kriegel, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. InProceedings of the Second International Conference on Knowledge Discovery and Data Mining, pages 226–231, August 1996

    1996

  19. [19]

    Open LLM Leaderboard, July 2025

    Hugging Face. Open LLM Leaderboard, July 2025. https://huggingface. co/spaces/open-llm-leaderboard/open llm leaderboard

    2025

  20. [20]

    Dimba: Transformer-Mamba diffusion models, June 2024

    Zhengcong Fei, Mingyuan Fan, et al. Dimba: Transformer-Mamba diffusion models, June 2024. arXiv:2406.01159

    Pith/arXiv arXiv 2024

  21. [21]

    Hafner Press, New York, reprinted edition, 1974

    Ronald A Fisher.The Design of Experiments. Hafner Press, New York, reprinted edition, 1974

    1974

  22. [22]

    Improved criteria for clustering based on the posterior similarity matrix.Bayesian Analysis, 4(2), June 2009

    Arno Fritsch and Katja Ickstadt. Improved criteria for clustering based on the posterior similarity matrix.Bayesian Analysis, 4(2), June 2009

    2009

  23. [23]

    Language Model Evaluation Harness: A framework for few-shot language model evaluation, July 2024

    Leo Gao, Jonathan Tow, et al. Language Model Evaluation Harness: A framework for few-shot language model evaluation, July 2024. https: //github.com/EleutherAI/lm-evaluation-harness

    2024

  24. [24]

    others Carlin

    Andrew Gelman and John B. others Carlin. Evaluating, comparing and expanding models. InBayesian Data Analysis, pages 165–197. Chapman and Hall, 3 edition, February 2025

    2025

  25. [25]

    Andrew Gelman, Xiao-Li Meng, and Hal S. Stern. Posterior predictive assessment of model fitness via realised discrepencies.Statistica Sinica, 6(4), October 1996

    1996

  26. [26]

    Measuring statistical dependence with Hilbert-Schmidt norms

    Arthur Gretton, Olivier Bousquet, et al. Measuring statistical dependence with Hilbert-Schmidt norms. InInternational Conference on Algorithmic Learning Theory, pages 63–77. Springer, October 2005

    2005

  27. [27]

    Mamba: Linear-time sequence modeling with selective state spaces, May 2024

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, May 2024. arXiv:2312.00752

    Pith/arXiv arXiv 2024

  28. [28]

    A survey on LLM-as-a-judge, March

    Jiawei Gu, Xuhui Jiang, et al. A survey on LLM-as-a-judge, March

  29. [29]

    DeepSeek-R1: Incentivizing reason- ing capability in LLMs via reinforcement learning, January 2025

    Daya Guo, Dejian Yang, et al. DeepSeek-R1: Incentivizing reason- ing capability in LLMs via reinforcement learning, January 2025. arXiv:2501.12948

    Pith/arXiv arXiv 2025

  30. [30]

    Scaling laws for neural language models, January 2020

    Jared Kaplan, Sam McCandlish, et al. Scaling laws for neural language models, January 2020. arXiv:2001.08361

    Pith/arXiv arXiv 2020

  31. [31]

    Jamba: Hybrid Transformer-Mamba language models

    Barak Lenz, Opher Lieber, et al. Jamba: Hybrid Transformer-Mamba language models. InThirteenth International Conference on Learning Representations, pages 67959–67984, March 2025

    2025

  32. [32]

    LLMs-as-judges: A comprehensive survey on LLM-based evaluation methods, December 2024

    Haitao Li, Qian Dong, et al. LLMs-as-judges: A comprehensive survey on LLM-based evaluation methods, December 2024. arXiv:2412.05579

    Pith/arXiv arXiv 2024

  33. [33]

    BC tribunal confirms companies remain liable for information provided by AI chatbot, February 2024

    Lisa R Lifshitz and Roland Hung. BC tribunal confirms companies remain liable for information provided by AI chatbot, February 2024. https://www.americanbar.org/groups/business law/resources/business- law-today/2024-february/bc-tribunal-confirms-companies-remain-liable- information-provided-ai-chatbot/

    2024

  34. [34]

    Least squares quantization in PCM.IEEE transactions on information theory, 28(2):129–137, 1982

    Stuart Lloyd. Least squares quantization in PCM.IEEE transactions on information theory, 28(2):129–137, 1982

    1982

  35. [35]

    A controlled study on long context exten- sion and generalization in LLMs, September 2024

    Yi Lu, Jing Nathan Yan, et al. A controlled study on long context exten- sion and generalization in LLMs, September 2024. arXiv:2409.12181

    Pith/arXiv arXiv 2024

  36. [36]

    HiBayES: A hierarchical Bayesian modeling framework for AI evaluation statistics, July 2025

    Lennart Luettgau, Harry Coppock, et al. HiBayES: A hierarchical Bayesian modeling framework for AI evaluation statistics, July 2025. arXiv:2505.05602

    Pith/arXiv arXiv 2025

  37. [37]

    Combining eigenvalues and variation of eigen- vectors for order determination.Biometrika, 103(4):875–887, 2016

    Wei Luo and Bing Li. Combining eigenvalues and variation of eigen- vectors for order determination.Biometrika, 103(4):875–887, 2016

    2016

  38. [38]

    Visualizing data using t-sne.Journal of machine learning research, 9(Nov):2579–2605, 2008

    Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of machine learning research, 9(Nov):2579–2605, 2008

    2008

  39. [39]

    Quantifying variance in evaluation benchmarks, June 2024

    Lovish Madaan, Aaditya K Singh, et al. Quantifying variance in evaluation benchmarks, June 2024. arXiv:2406.10229

    Pith/arXiv arXiv 2024

  40. [40]

    On evaluating adversarial robustness of volumetric medical segmentation models, September 2024

    Hashmat Shadab Malik, Numan Saeed, et al. On evaluating adversarial robustness of volumetric medical segmentation models, September 2024. arXiv:2406.08486

    Pith/arXiv arXiv 2024

  41. [41]

    Towards evaluat- ing the robustness of visual state space models, September 2024

    Hashmat Shadab Malik, Fahad Shamshad, et al. Towards evaluat- ing the robustness of visual state space models, September 2024. arXiv:2406.09407

    Pith/arXiv arXiv 2024

  42. [42]

    Inadequacies of large language model benchmarks in the era of generative artificial intelligence, October

    Timothy R McIntosh, Teo Susnjak, et al. Inadequacies of large language model benchmarks in the era of generative artificial intelligence, October

  43. [43]

    Frontier models are capable of in-context scheming, January 2025

    Alexander Meinke, Bronson Schoen, et al. Frontier models are capable of in-context scheming, January 2025. arXiv:2412.04984

    Pith/arXiv arXiv 2025

  44. [44]

    ShortGPT: Layers in large language models are more redundant than you expect, October 2024

    Xin Men, Mingyu Xu, et al. ShortGPT: Layers in large language models are more redundant than you expect, October 2024. arXiv:2403.03853

    Pith/arXiv arXiv 2024

  45. [45]

    Adding error bars to evals: A statistical approach to language model evaluations, November 2024

    Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations, November 2024. arXiv:2411.00640

    Pith/arXiv arXiv 2024

  46. [46]

    Accuracy on the line: On the strong correlation between out-of-distribution and in-distribution generaliza- tion

    John P Miller, Rohan Taori, et al. Accuracy on the line: On the strong correlation between out-of-distribution and in-distribution generaliza- tion. InInternational conference on machine learning, pages 7721–7735, July 2021

    2021

  47. [47]

    Mistral Small 3, January 2025

    Mistral. Mistral Small 3, January 2025. https://mistral.ai/news/ mistral-small-3

    2025

  48. [48]

    Evaluating the evaluator: Measuring LLMs’ adherence to task evaluation instructions, August 2024

    Bhuvanashree Murugadoss, Christian Poelitz, et al. Evaluating the evaluator: Measuring LLMs’ adherence to task evaluation instructions, August 2024. arXiv:2408.08781

    arXiv 2024

  49. [49]

    HyenaDNA: Long-range genomic sequence modeling at single nucleotide resolution, November 2023

    Eric Nguyen, Michael Poli, et al. HyenaDNA: Long-range genomic sequence modeling at single nucleotide resolution, November 2023. arXiv:2306.15794v2

    Pith/arXiv arXiv 2023

  50. [50]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations, April 2024. arXiv:2404.13076

    Pith/arXiv arXiv 2024

  51. [51]

    Distillation-based cross- model transferable adversarial attack for remote sensing image classifi- cation.Remote Sensing, 17(10):1700, January 2025

    Xiyu Peng, Jingyi Zhou, and Xiaofeng Wu. Distillation-based cross- model transferable adversarial attack for remote sensing image classifi- cation.Remote Sensing, 17(10):1700, January 2025

    2025

  52. [52]

    Exploring adversarial robustness of deep state space models, October 2024

    Biqing Qi, Yang Luo, et al. Exploring adversarial robustness of deep state space models, October 2024. arXiv:2406.05532

    Pith/arXiv arXiv 2024

  53. [53]

    Disentangled representation learning with large language models for text-attributed graphs, March 2024

    Yijian Qin, Xin Wang, et al. Disentangled representation learning with large language models for text-attributed graphs, March 2024. arXiv:2310.18152

    Pith/arXiv arXiv 2024

  54. [54]

    Pretrained models - sentence Transformers, March 2025

    Nils Reimers and Iryna Gurevych. Pretrained models - sentence Transformers, March 2025. https://sbert.net/docs/sentence transformer/ pretrained models.html

    2025

  55. [55]

    Springer, 2nd edition, July 2004

    Christian B Robert and George Casella.Monte Carlo integration. Springer, 2nd edition, July 2004

    2004

  56. [56]

    Clustering algorithms: A comparative approach.Public Library of Science one, 14(1), January 2019

    Mayra Z Rodriguez, Cesar H Comin, et al. Clustering algorithms: A comparative approach.Public Library of Science one, 14(1), January 2019

    2019

  57. [57]

    Textual bayes: Quantifying uncertainty in LLM-based systems, June 2025

    Brendan Leigh Ross, No ¨el V ouitsis, et al. Textual bayes: Quantifying uncertainty in LLM-based systems, June 2025. arXiv:2506.10060

    Pith/arXiv arXiv 2025

  58. [58]

    NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark, December 2023

    Oscar Sainz, Jon Ander Campos, et al. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark, December 2023. arXiv:2310.18018

    Pith/arXiv arXiv 2023

  59. [59]

    all-MiniLM-L6-v2, January 2024

    Sentence-Transformers. all-MiniLM-L6-v2, January 2024. https:// huggingface.co/sentence-transformers/all-MiniLM-L6-v2

    2024

  60. [60]

    Mamba-2.8b, March 2024

    State-spaces. Mamba-2.8b, March 2024. https://huggingface.co/ state-spaces/mamba-2.8b-hf

    2024

  61. [61]

    Limitations of the LLM-as-a- judge approach for evaluating LLM outputs in expert knowledge tasks

    Annalisa Szymanski, Noah Ziems, et al. Limitations of the LLM-as-a- judge approach for evaluating LLM outputs in expert knowledge tasks. InProceedings of the 30th International Conference on Intelligent User Interfaces, pages 952–966, March 2025

    2025

  62. [62]

    Semi-supervised and un-supervised clustering: A review and experimental evaluation.Information Systems, 114:102178, Febru- ary 2023

    Kamal Taha. Semi-supervised and un-supervised clustering: A review and experimental evaluation.Information Systems, 114:102178, Febru- ary 2023

    2023

  63. [63]

    Hierarchical Dirchlet processes

    Yee Whye Teh, Michael I Jordan, et al. Hierarchical Dirchlet processes. Journal of the American Statistical Association, 101(476), January 2012

    2012

  64. [64]

    Who belongs in the family?Psychometrika, 18(4):267–276, December 1953

    Robert L Thorndike. Who belongs in the family?Psychometrika, 18(4):267–276, December 1953

    1953

  65. [65]

    Towards greater leverage: Scaling laws for efficient mixture-of-experts language models, August 2025

    Changxin Tian, Kunlong Chen, et al. Towards greater leverage: Scaling laws for efficient mixture-of-experts language models, August 2025. arXiv:2507.17702

    arXiv 2025

  66. [66]

    Bayesian prompt ensembles: Model uncertainty estimation for black-box large language models

    Francesco Tonolini, Nikolaos Aletras, et al. Bayesian prompt ensembles: Model uncertainty estimation for black-box large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 12229–12272, August 2024

    2024

  67. [67]

    A field guide to automatic evaluation of LLM-generated summaries

    Tempest A van Schaik and Brittany Pugh. A field guide to automatic evaluation of LLM-generated summaries. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2832–2836, July 2024

    2024

  68. [68]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, et al. Attention is all you need. In Advances in Neural Information Processing Systems 30, December 2017

    2017

  69. [69]

    Bayesian cluster analysis.Philosophical Transactions of the Royal Statistical Society A, 381(2247), March 2023

    Sara Wade. Bayesian cluster analysis.Philosophical Transactions of the Royal Statistical Society A, 381(2247), March 2023

    2023

  70. [70]

    An empirical study of Mamba- based language models, June 2024

    Roger Waleffe, Wonmin Byeon, et al. An empirical study of Mamba- based language models, June 2024. arXiv:2406.07887

    Pith/arXiv arXiv 2024

  71. [71]

    Confidence in large language model evalu- ation: a Bayesian approach to limited-sample challenges, April 2025

    Xiao Xiao, Yu Su, et al. Confidence in large language model evalu- ation: a Bayesian approach to limited-sample challenges, April 2025. arXiv:2504.21303

    Pith/arXiv arXiv 2025

  72. [72]

    Benchmarking benchmark leakage in large language models, April 2024

    Ruijie Xu, Zengzhi Wang, et al. Benchmarking benchmark leakage in large language models, April 2024. arXiv:2404.18824

    Pith/arXiv arXiv 2024

  73. [73]

    PlainMamba: Improv- ing non-hierarchical Mamba in visual recognition, August 2024

    Chenhongyi Yang, Zehui Chen, et al. PlainMamba: Improv- ing non-hierarchical Mamba in visual recognition, August 2024. arXiv:2403.17695

    Pith/arXiv arXiv 2024

  74. [74]

    Rethinking benchmark and contam- ination for language models with rephrased samples, November 2023

    Shuo Yang, Wei-Lin Chiang, et al. Rethinking benchmark and contam- ination for language models with rephrased samples, November 2023. arXiv:2311.04850

    Pith/arXiv arXiv 2023

  75. [75]

    LessLeak-Bench: A first investigation of data leakage in LLMs across 83 software engineering benchmarks, February 2025

    Xin Zhou, Martin Weyssow, et al. LessLeak-Bench: A first investigation of data leakage in LLMs across 83 software engineering benchmarks, February 2025. arXiv:2502.06215

    Pith/arXiv arXiv 2025

  76. [76]

    On the role of attention heads in large language model safety, February 2025

    Zhenhong Zhou, Haiyang Yu, et al. On the role of attention heads in large language model safety, February 2025. arXiv:2410.13708

    Pith/arXiv arXiv 2025

  77. [77]

    What’s terminal ANSI code to render green terminal text?

    Mu Zhu and Ali Ghodsi. Automatic dimensionality selection from the scree plot via the use of profile likelihood.Computational Statistics & Data Analysis, 51(2):918–930, November 2006. APPENDIX A. Supplementary results for the independence assumption investigations in Section IV-A Fig. 5: Visualisation of the prompts from each attack. From left to right: (...

    2006