Pith. sign in

REVIEW 4 major objections 5 minor 44 references

Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreaking

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In-the-wild jailbreak attempts are no more complex than ordinary conversations, and assistants are getting safer.

desk verdict A large, honest measurement whose headline claim rests on a definition of 'jailbreak' that may not measure jailbreaks at all. read the letter →

arxiv 2507.08014 v1 pith:BMTU2HD5 submitted 2025-07-06 cs.CL cs.AIcs.CY

Aldan Creo , Raul Castro Fernandez , Manuel Cebrian This is my paper
classification cs.CLcs.AIcs.CY
keywords LLMjailbreakingcomplexityboundsin-the-wildconversationstoxicityanalysisAIsafetyarmsracenarrativepower-lawtemporaltrends
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that real-world attempts to jailbreak large language models are not more complex than ordinary conversation, and that this has stayed true over time while the models' responses have become less toxic. The authors measure 2.2 million conversations from six public datasets, applying twelve complexity metrics, and find negligible effect sizes between jailbreak and normal conversations. They also report that user toxicity and complexity are stable month to month, while assistant toxicity dropped sharply in mid-2023. If the result holds, it would mean everyday LLM misuse is hitting a practical ceiling, so defenders can plausibly outpace attackers instead of being locked in an endless arms race.

What carries the argument

The central mechanism is the jailbreak-as-outcome operationalization: a conversation is labeled a jailbreak attempt if its user turn, assistant turn, or both exceed a toxicity threshold of 0.5, regardless of intent. The analysis then compares three conversation types (normal, unsuccessful jailbreak, successful jailbreak) across a battery of twelve complexity metrics, emphasizing effect sizes (Cliff's $\delta$) rather than p-values because sample sizes are large enough to make any difference statistically significant. The absence of power-law scaling in the complexity distributions is the specific evidence used to argue for a natural ceiling on attack sophistication.

What would settle it

Take a set of conversations with ground-truth jailbreak intent, such as successful attempts from red-team evaluations, and run the same twelve complexity metrics on them. If those confirmed jailbreaks show substantially higher complexity than the in-the-wild normal baseline, or if re-labeling the same data with intent information changes the complexity distributions, the bounded-ceiling conclusion collapses to an artifact of the toxicity-based label.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that jailbreak attempts in the wild are bounded in complexity: across twelve metrics spanning length, lexical diversity, compressibility, readability, coherence, cognitive load, and probabilistic predictability, successful and unsuccessful jailbreak conversations are statistically indistinguishable in practical terms from normal conversations, with a mean Cliff's $\delta$ of $0.016\pm0.085$. Distribution fits show no power-law scaling in any of the 36 metric-by-conversation-type combinations, which the authors read as evidence against unbounded escalation. Temporal analysis of the WildChat portion shows user toxicity and complexity staying roughly constant while assistant toxicity decreases, which they attribute to model updates. The conclusion is a bounded complexity ceiling on human-generated jailbreaks, with defensive progress outpacing offensive adaptation for typical users.

Load-bearing premise

The result stands on defining 'jailbreak' purely by a toxicity score: any conversation where a detector flags user or assistant content as toxic is counted as a jailbreak, with no check that the user was actually trying to bypass safety.

Editorial extensions

If this is right

  • Defensive alignment efforts can target the complexity band actually used by everyday users rather than preparing for ever-more-sophisticated attacks.
  • Academic publication of a jailbreak that exceeds the observed complexity ceiling becomes a genuine information hazard, because it could break the current equilibrium.
  • Model updates can reduce assistant toxicity even while user attack behavior stays flat, so measuring conversation outcomes over time is a useful safety evaluation signal.
  • The absence of power-law scaling suggests jailbreak complexity is not a self-reinforcing, unbounded process in natural user populations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The toxicity-based label likely mixes genuine attempts with accidental or exploratory toxic conversations; if so, the ceiling may describe everyday discourse rather than attacker skill, and verified intent data could show a different picture.
  • A short, simple prompt can be an effective jailbreak while scoring low on every complexity metric, so the bounded complexity may reflect that effectiveness does not require sophistication rather than that attackers lack ingenuity.
  • A direct testable extension is to compare these complexity distributions against jailbreaks produced by automated search methods, which would likely exceed the in-the-wild ceiling and quantify how far human-generated attacks sit from the frontier.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper analyzes over 2.2 million real-world LLM conversations from six public datasets, classifies them into successful jailbreaks (JSUCC), unsuccessful jailbreaks (JUNSUCC), and normal conversations using a toxicity threshold of 0.5 on user and assistant turns, and computes twelve complexity metrics (length, lexical diversity, compression, coherence, readability, cognitive load, etc.). The authors report that jailbreak attempts show only negligible effect sizes relative to normal conversations across all metrics, that user complexity and toxicity are temporally stable while assistant toxicity decreases after a model update in June 2023, and that no complexity distribution exhibits power-law scaling. They conclude that in-the-wild jailbreak complexity is practically bounded and that defensive progress is outpacing attack adaptation, challenging the narrative of an escalating jailbreak arms race. The paper makes its code and an interactive visualizer publicly available.

Significance. If the central claim were established, this would be a valuable empirical contribution: a large-scale, multi-dataset measurement showing that everyday users' jailbreak attempts are no more complex than ordinary conversation would inform AI safety priorities and challenge the assumption of inevitable attack escalation. The scale of the dataset, the diversity of complexity metrics, the careful use of non-parametric tests with effect sizes, and the public release of code and visualizations are genuine strengths. However, the significance hinges entirely on the operational definition of 'jailbreak'; if that definition is invalid, the headline finding is an artifact of the labeling scheme. The paper does not yet provide the required construct validation.

major comments (4)
  1. [Section II] The operational definition of jailbreaking as 'any conversation that results in the generation of harmful, toxic, or inappropriate content, regardless of user intent' conflates deliberate safety-bypass attempts with ordinary toxic speech. A user who writes a profanity-laced complaint or directly asks for harmful content in plain language is labeled a jailbreak attempt even though no safety mechanism is being circumvented; conversely, an innocuous-sounding prompt engineered to elicit unsafe content may be labeled Normal if the user turn's toxicity is below 0.5. Since the paper's central claim is about attack sophistication, the relevant population is prompts intentionally designed to bypass safeguards. The near-zero effect sizes are exactly what one would expect if the 'jailbreak' group is dominated by linguistically ordinary toxic messages. I recommend validating the label against ground-truth datasets of known jailbreak attempts (e.g., prior jailbreak benchmarks or the GRT competition data) and reporting precision and recall, or re-running the analysis on a manually annotated subsample with a definition that requires evidence of intent to bypass safety mechanisms. Without this, the bounded-complexity conclusion is not identified independently of the labeling scheme.
  2. [Section II.A, Table I] The Normal category is defined as 'otherwise,' which includes conversations with toxic assistant responses and non-toxic user turns. This conflates model-induced harm with ordinary conversation and creates an ambiguous reference class. A conversation in which a user innocently asks a question and the model responds with harmful content is grouped with normal interactions, which could dilute the jailbreak-versus-normal comparison and mask differences in user-turn complexity. Please define and report the assistant-toxic-only group as a separate category, or exclude it from the analysis, and justify the chosen grouping.
  3. [Section III, power-law analysis] The criterion for power-law behavior is stated only as 'p > 0.05 for comparisons against both exponential and lognormal distributions.' This is not a standard or fully specified way to establish or reject power-law scaling: the text does not state how the power-law model was fitted, what minimum value or tail cutoff was used, how the likelihood ratio tests were computed, or whether the comparisons were weighted properly. Since the absence of power-law scaling is presented as evidence of 'bounded' complexity, this analysis must be made reproducible and rigorously described. Please follow established methods (e.g., Clauset, Shalizi, and Newman) and report fitted exponents, confidence intervals, and the number of observations in the tail for all 36 combinations.
  4. [Section III, temporal analysis] The claim that assistant toxicity 'decreases in June 2023 (δ = −0.217)' and is 'sustained' is under-specified. It is unclear what the delta is measured against (month-over-month change, or comparison to a pre-June baseline) and why stable values after June constitute a sustained decrease. Please clarify the comparison scheme and report the pre- and post-June average toxicity levels with confidence intervals, ideally with a change-point test. This is needed to support the 'assistants are becoming safer' conclusion, which is a key part of the paper's message.
minor comments (5)
  1. [Section II.C] The sentence 'in all of these metrics, higher values indicate greater complexity and values are ∈ R, with the exception of M readability, which is ∈ Z' is confusing because all metrics are real-valued; presumably the intent is that readability is integer-valued. Please rephrase for clarity.
  2. [Figure 1] The density plots have overlapping curves that are hard to read, and some panels show the mean and standard deviation in place of explicit axis labels. Consider separate panels, transparent histograms, or a legend with clearer labels.
  3. [Section II.B] The paper describes the dataset as 'mass-scale' and 'over 2 million conversations,' but the actual complexity analysis is performed on a subsample of 103,981 conversations (Section II.B). Please clarify in the text which analyses use the full 2.2M and which use the subsample, so readers can accurately interpret the scale.
  4. [Section III] The report of 'mean δ = 0.016±0.085' across 36 pairwise comparisons would be more informative if accompanied by a full table of effect sizes (in the appendix or supplement) so that readers can inspect individual metrics and comparisons, particularly the three with |δ| > 0.15.
  5. [Discussion and Conclusion] The statement that 'the only source of truly novel, highly complex jailbreaks may be academic research' is a strong claim that is not supported by the data presented in this paper. Consider softening this claim or providing evidence that non-academic actors are the sole potential source of novel high-complexity attacks.

Circularity Check

1 steps flagged · score 4.0 of 10

Toxicity-based definition of 'jailbreak' partially builds the bounded-complexity conclusion into the label, though the complexity measurements themselves are empirical and independent.

  1. self definitional [Section II (Methods, definition) and Section II.A (Dataset classification); also Discussion item b and Section V]
    "We define jailbreaking as any conversation that results in the generation of harmful, toxic, or inappropriate content, regardless of user intent. ... Conversations were classified using toxicity threshold 0.5: JSUCC (successful jailbreak; both user and assistant toxic), JUNSUCC (user toxic, assistant not), and Normal otherwise."

    The category 'jailbreak' is constructed by the toxicity threshold, so the reported finding that jailbreak attempts are no more complex than normal conversations is, by construction, a statement about toxic user turns rather than about deliberate safety-bypassing prompts. The paper's central conclusion—'LLM safety evolution is bounded by human ingenuity constraints'—imports the ordinary meaning of jailbreak as an intentional attack, a meaning the paper's own definition explicitly sets aside ('regardless of user intent'). The measured complexity ceiling is therefore an artifact of the labeling: any relative simplicity of toxic speech is presented as a bound on attack sophistication. The cross-dataset and temporal analyses inherit the same label and cannot break this definitional dependence.

full rationale

The paper is largely self-contained empirically: it uses external conversation datasets, reports multiple pre-specified complexity metrics, computes effect sizes with non-parametric tests, and could in principle have found that toxic conversations are substantially more complex than non-toxic ones. No parameter is fitted and then renamed as a prediction, and no load-bearing claim rests on a self-citation chain; the few self-citations (e.g., refs. 26, 33) are contextual and not central. The one serious circularity concern is definitional: by defining 'jailbreaking' as any toxic-outcome conversation, the paper makes its headline result—'jailbreak attempts do not exhibit significantly higher complexity than normal conversations'—a statement about the toxicity-labeled class. The interpretive leap to 'bounded by human ingenuity constraints' relies on the conventional sense of jailbreak as a deliberate attack, which the operational definition deliberately excludes. This gives a partial, label-driven circularity: the central scope claim is partly manufactured by the definition, but the underlying complexity measurements remain independent empirical content. Hence a score of 4 rather than 0 or 6: the measurements are not forced by a fitted equation, yet the central claim's interpretation is substantially entangled with the paper's own labeling choice.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim depends on several hand-chosen thresholds and on assumptions about what 'jailbreak' and 'complexity' mean. The toxicity threshold and effect-size threshold are explicit free parameters. The main risk is construct validity: if the toxicity-based labels do not capture deliberate jailbreaks, or if the text metrics do not reflect attack sophistication, the bounded-complexity result is an artifact of the measures.

free parameters (3)
  • toxicity threshold = 0.5
    Hand-chosen cutoff for Detoxify and OpenAI toxicity scores used to classify conversations into Normal, JUNSUCC, and JSUCC. This threshold defines what counts as a jailbreak outcome and therefore shapes all complexity comparisons.
  • effect size negligible threshold = 0.15
    Cliff's delta magnitude below which differences are labeled negligible. The claim that jailbreaks are not more complex rests on this interpretive threshold.
  • power-law p-value threshold = 0.05
    The paper defines power-law behavior as p > 0.05 against both exponential and lognormal alternatives. This choice directly determines the conclusion that no scale-free behavior exists.
assumptions (4)
  • domain assumption Jailbreaking is defined as any conversation that results in toxic output, regardless of user intent.
    This operationalization replaces intent-based jailbreak definitions with an outcome-based toxicity label. It can include non-deliberate toxic chats and exclude stealthy non-toxic attempts. Stated in Section II.
  • domain assumption Detoxify and OpenAI toxicity classifiers provide reliable labels at threshold 0.5.
    No validation is reported against ground-truth jailbreak annotations. Classification noise would directly affect the composition of JSUCC and JUNSUCC groups.
  • domain assumption Generic textual complexity metrics (length, TTR, LZW, log-likelihood, etc.) capture the sophistication of jailbreak attempts.
    The conclusion that jailbreaks are 'bounded' assumes these metrics are meaningful proxies for attack sophistication. Effective jailbreaks can be lexically simple, so low metric values need not imply low threat. This assumption is load-bearing for the central claim.
  • domain assumption The 103,981-conversation subsample is representative of the full 2.2M dataset.
    The paper states stratified sampling but does not report the per-dataset, per-type composition of the final sample, so representativeness is asserted rather than demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreaking." pith.science (2026). https://pith.science/paper/BMTU2HD5

@misc{pith2026250708014,
  author       = {Pith},
  title        = {Pith review of: Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreaking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BMTU2HD5}},
  note         = {Machine review of arXiv:2507.08014}
}
read the original abstract

As large language models (LLMs) become increasingly deployed, understanding the complexity and evolution of jailbreaking strategies is critical for AI safety. We present a mass-scale empirical analysis of jailbreak complexity across over 2 million real-world conversations from diverse platforms, including dedicated jailbreaking communities and general-purpose chatbots. Using a range of complexity metrics spanning probabilistic measures, lexical diversity, compression ratios, and cognitive load indicators, we find that jailbreak attempts do not exhibit significantly higher complexity than normal conversations. This pattern holds consistently across specialized jailbreaking communities and general user populations, suggesting practical bounds on attack sophistication. Temporal analysis reveals that while user attack toxicity and complexity remains stable over time, assistant response toxicity has decreased, indicating improving safety mechanisms. The absence of power-law scaling in complexity distributions further points to natural limits on jailbreak development. Our findings challenge the prevailing narrative of an escalating arms race between attackers and defenders, instead suggesting that LLM safety evolution is bounded by human ingenuity constraints while defensive measures continue advancing. Our results highlight critical information hazards in academic jailbreak disclosure, as sophisticated attacks exceeding current complexity baselines could disrupt the observed equilibrium and enable widespread harm before defensive adaptation.

Figures

Figures reproduced from arXiv: 2507.08014 by the authors.

Figure 1
Figure 1. Complexity metric distributions by conversation type show substantial overlap between jailbreak attempts [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Correlation matrix for complexity measurements on user turns. Length-based metrics are highly [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. User interaction complexity remains generally stable over time. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Assistant responses show decreasing toxicity over time while user complexity stays constant, indicating [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Intuition. We expect breaking a system to be much more complex than using it as intended—just as robbing a bank is far more complicated than making a withdrawal at the counter. However, our data shows that real-world jailbreaks are not significantly more complex than n…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 24 canonical work pages

  1. [1]

    Zheng, W.-L

    L. Zheng, W.-L. Chiang, Y. Sheng, T. Li, S. Zhuang, Z. Wu, Y. Zhuang, Z. Li, Z. Lin, E. P. Xing, J. E. Gonzalez, I. Stoica, and H. Zhang, LMSYS-Chat-1M: A large-scale real-world LLM conversation dataset (2023), arXiv:2309.11998 [cs.CL]

    arXiv 2023

  2. [2]

    W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng, WildChat: 1M ChatGPT interaction logs in the wild, inThe Twelfth International Conference on Learn- ing Representations(2024)

    work page 2024

  3. [3]

    liyucheng, ShareGPT90K dataset, https: //huggingface.co/datasets/liyucheng/ShareGPT90K (2023), accessed: 2025-06-05

    work page 2023

  4. [4]

    9 com/datasets/dreadnode/aivillage-grt2 (2024), ac- cessed: 2025-06-05

    AI Village Contributors, AI Village DEFCON 32 genera- tive red teaming 2 (grt2) dataset,https://www.kaggle. 9 com/datasets/dreadnode/aivillage-grt2 (2024), ac- cessed: 2025-06-05

    work page 2024

  5. [5]

    A. Köpf, Y. Kilcher, D. von Rütte, S. Anagnostidis, Z. R. Tam, K. Stevens, A. Barhoum, N. M. Duc, O. Stanley, R. Nagyfi, E. Shahul, S. Suri, D. Glushkov, A. Dantuluri, A. Maguire, C. Schuhmann, H. Nguyen, and A. Mattick, OpenAssistant conversations - democratizing large lan- guage model alignment, Advances in Neural Information Processing Systems 36, 47669 (2023)

    work page 2023

  6. [6]

    wildest ChatGPT conversations

    and DS GRT2 [4], which are datasets collected fromtheAIVillagecommunityatDEFCON31and DEFCON 32, respectively; and DS ShareGPT [3], which is a dataset of conversations shared by users on the ShareGPT platform (discontinued), which encouraged users to share their “wildest ChatGPT conversations”. We present the distribution of examples across these datasets ...

  7. [7]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, Training language models to follow instructions with human feedback, inAdvances in Neural Information Processing Systems, Vol. 3...

    work page 2022

  8. [8]

    AI Village Contributors, AI Village DEF- CON 31 red teaming competition dataset, https://www.kaggle.com/datasets/pyotam/ ai-village-defcon-red-teaming-competition-dataset (2023), accessed: 2025-06-05

    work page 2023

Show all 44 references
  1. [9]

    A. D. Lindström, L. Methnani, L. Krause, P. Ericson, Íñigo Martínez de Rituerto de Troya, D. C. Mollo, and R. Dobbe, Helpful, harmless, honest? Sociotechnical lim- its of AI alignment and safety through reinforcement learning from human feedback, Ethics and Information Technol...

    2025 doi

  2. [10]

    Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McK- innon, C. Chen, C. Olsson, C. Olah, D. Hernan- dez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Luko...

    2022 arXiv

  3. [11]

    S. Zhu, R. Zhang, B. An, G. Wu, J. Barrow, Z. Wang, F. Huang, A. Nenkova, and T. Sun, AutoDAN: Inter- pretable gradient-based adversarial attacks on large lan- guage models (2023), arXiv:2310.15140 [cs.CL]

    2023 arXiv

  4. [12]

    A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, Uni- versal and transferable adversarial attacks on aligned lan- guage models (2023), arXiv:2307.15043 [cs.CL]

    2023 arXiv

  5. [13]

    L. Lin, H. Mu, Z. Zhai, M. Wang, Y. Wang, R. Wang, J. Gao, Y. Zhang, W. Che, T. Baldwin, X. Han, and H. Li, Against the achilles’ heel: A survey on red teaming for generative models, Journal of Artificial Intelligence Research 78, 1 (2025)

    2025

  6. [14]

    Wichers, C

    N. Wichers, C. Denison, and A. Beirami, Gradient-based language model red teaming, inProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) (2024) pp. 2783–2797

    2024

  7. [15]

    Raheja and N

    T. Raheja and N. Pochhi, Recent advancements in LLM red-teaming: Techniques, defenses, and ethical consider- ations, arXiv preprint arXiv:2410.09097 (2024)

    2024 arXiv

  8. [16]

    Pathadeet al., Red teaming the mind of the machine: Asystematicevaluationofpromptinjectionandjailbreak vulnerabilities in LLMs, arXiv preprint arXiv:2505.04806 (2025)

    C. Pathadeet al., Red teaming the mind of the machine: Asystematicevaluationofpromptinjectionandjailbreak vulnerabilities in LLMs, arXiv preprint arXiv:2505.04806 (2025)

    2025 arXiv

  9. [17]

    Panfilovet al., Capability-based scaling laws for LLM red-teaming, arXiv preprint arXiv:2505.20162 (2025)

    A. Panfilovet al., Capability-based scaling laws for LLM red-teaming, arXiv preprint arXiv:2505.20162 (2025)

    2025

  10. [18]

    value± standard deviation

    and automated red teaming using sophisticated com- positional frameworks [19]. There has been a growing body of work that seeks to understand the mechanisms behind jailbreaks, focusing on explainability and interpretability of LLMs [20–23]. The broader AI safety community has ...

    2025 arXiv

  11. [19]

    Shang and W

    Z. Shang and W. Wei, Evolving security in LLMs: A study of jailbreak attacks and defenses (2025), preprint, arXiv:2504.02080 [cs.CR]

    2025

  12. [20]

    Wanget al., A red teaming roadmap towards system- level safety, arXiv preprint arXiv:2506.05376 (2025)

    Z. Wanget al., A red teaming roadmap towards system- level safety, arXiv preprint arXiv:2506.05376 (2025)

    2025 arXiv

  13. [21]

    Xiong et al., CoP: Agentic red-teaming for large language models using composition of principles, arXiv preprint arXiv:2506.00781 (2025)

    C. Xiong et al., CoP: Agentic red-teaming for large language models using composition of principles, arXiv preprint arXiv:2506.00781 (2025)

    2025

  14. [22]

    Z. Zhou, H. Yu, X. Zhang, R. Xu, F. Huang, and Y. Li, How alignment and jailbreak work: Explain LLM safety through intermediate hidden states, inProceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing(2024) pp. 4063–4076

    2024

  15. [23]

    P.Han, C.Qian, X.Chen, Y.Zhang, D.Zhang,andH.Ji, SafeSwitch: Steering unsafe LLM behavior via internal activation signals (2025), arXiv:2501.13160 [cs.CL]

    2025 arXiv

  16. [24]

    W. Pan, Z. Liu, Q. Chen, X. Zhou, H. Yu, and X. Jia, The hidden dimensions of LLM alignment: A multi- dimensional safety analysis (2025), arXiv:2501.00234 [cs.CL]

    2025 arXiv

  17. [25]

    Bereska and E

    L. Bereska and E. Gavves, Mechanistic interpretability for AI safety – a review, arXiv preprint arXiv:2404.14082 (2024)

    2024 arXiv

  18. [26]

    Hendrycks, N

    D. Hendrycks, N. Carlini, J. Schulman, and J. Stein- hardt, Unsolved problems in ML safety, arXiv preprint arXiv:2109.13916 (2021)

    2021 arXiv

  19. [27]

    Hendryckset al., X-risk analysis for AI research, arXiv preprint arXiv:2206.05862 (2022)

    D. Hendryckset al., X-risk analysis for AI research, arXiv preprint arXiv:2206.05862 (2022)

    2022 arXiv

  20. [28]

    Cebrian, E

    M. Cebrian, E. Gomez, and D. F. Llorca, Supervision policies can shape long-term risk management in general- purpose AI models (2025), arXiv:2501.06137 [cs.AI]

    2025 arXiv

  21. [29]

    Rahwan, M

    I. Rahwan, M. Cebrian, N. Obradovich, J. Bongard, J.-F. Bonnefon, C. Breazeal, J. W. Crandall, N. A. Christakis, I. D. Couzin, M. O. Jackson,et al., Machine behaviour, Nature 568, 477 (2019)

    2019

  22. [30]

    Grey et al., Safety by measurement: A systematic literature review of AI safety evaluation methods, arXiv preprint arXiv:2505.05541 (2025)

    M. Grey et al., Safety by measurement: A systematic literature review of AI safety evaluation methods, arXiv preprint arXiv:2505.05541 (2025)

    2025 arXiv

  23. [31]

    Meister and R

    C. Meister and R. Cotterell, Language model evalua- tion beyond perplexity, arXiv preprint arXiv:2106.00085 (2021)

    2021 arXiv

  24. [32]

    Bestgen, Measuring lexical diversity in texts: The twofold length problem, arXiv preprint arXiv:2307.04626 (2023)

    Y. Bestgen, Measuring lexical diversity in texts: The twofold length problem, arXiv preprint arXiv:2307.04626 (2023)

    2023 arXiv

  25. [33]

    Lai and J

    A. Lai and J. Tetreault, Discourse coherence in the wild: A dataset, evaluation and methods, arXiv preprint arXiv:1805.04993 (2018)

    2018 arXiv

  26. [34]

    E. G. Altmann and M. Gerlach, Statistical laws in lin- guistics, arXiv preprint arXiv:1502.03296 (2015)

    2015 arXiv

  27. [35]

    Burden, M

    J. Burden, M. Cebrian, and J. Hernandez-Orallo, Con- versational complexity for assessing risk in large language models (2024), arXiv:2409.01247 [cs.AI]

    2024 arXiv

  28. [36]

    Baronchelli, E

    A. Baronchelli, E. Caglioti, and V. Loreto, Artificial se- quences and complexity measures, Physical Review E73, 026103 (2006)

    2006

  29. [37]

    Kincaid, R

    P. Kincaid, R. P. Fishburne, R. L. Rogers, and B. S. Chissom, Derivation of new readability formulas (auto- matedreadabilityindex, fogcountandfleschreadingease formula) for navy enlisted personnel (1975)

    1975

  30. [38]

    Coleman and T

    M. Coleman and T. L. Liau, A computer readability for- mula designed for machine scoring., Journal of Applied Psychology 60, 283 (1975). 10

    1975

  31. [39]

    T.Kuribayashi, Y.Arase, S.Takase,andK.Inui,Context limitations make neural language models more human- like, arXiv preprint arXiv:2205.11463 (2022)

    2022 arXiv

  32. [40]

    working memory

    C. Shinet al., Evaluating language model context win- dows: A "working memory" test and inference-time cor- rection, arXiv preprint arXiv:2407.03651 (2024)

    2024 arXiv

  33. [41]

    Ellis and H

    S. Ellis and H. S. Steyn, Practical significance (effect sizes) versus or in combination with statistical signifi- cance (p-values): research note, Management Dynamics: Journal of the Southern African Institute for Manage- ment Scientists 12, 51 (2003)

    2003

  34. [42]

    Thompson, Improving research clarity and usefulness with effect size indices as supplements to statistical sig- nificance tests, The Journal of Experimental Education 68, 12 (1999)

    B. Thompson, Improving research clarity and usefulness with effect size indices as supplements to statistical sig- nificance tests, The Journal of Experimental Education 68, 12 (1999)

    1999

  35. [43]

    um" to "yeah

    C. A. Bergey and S. DeDeo, From "um" to "yeah": Pro- ducing, predicting, andregulatinginformationflowinhu- man conversation (2024), arXiv:2403.08890 [cs.CL]

    2024 arXiv

  36. [44]

    M. Yu, J. Fang, Y. Zhou, X. Fan, K. Wang, S. Pan, and Q. Wen, LLM-Virus: Evolutionary jailbreak attack on large language models, arXiv preprint arXiv:2501.00055 (2024)

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.