{"id":"8832b9cd-3022-46ea-9787-3855cb0ab121","arxiv_id":"2505.02273","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Optimized prompts use rare, punctuation-heavy tokens and produce activation patterns that simple classifiers can distinguish from natural language, with similar layer-wise dynamics across model families.","lead":"Researchers analyzed machine-generated 'optimized' prompts, which are gibberish strings that can steer language models, and found they rely on rare words and punctuation while producing distinctive internal neural patterns. The work may help detect jailbreak-style inputs before they fully affect model behavior.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never reports the sample size n in the KL estimator (Eq. 1); if n is small, the dKL<=10 filter and all influence-score rankings that support the central claims could be noise.","rationale":"Reader's weakest assumption is the same one I would flag: the KL estimator's finite-sample behavior. I agree, with a small emphasis shift: the issue is not just the unreported n but the reuse of the same sampled continuations for selection and evaluation, which lets GCG overfit the dKL estimate and makes the dKL <= 10.0 filter a noisy selection rule. This matters because every quantitative claim in the paper is downstream of Eq. 1: which tokens are influential, what the optimized set looks like, and how representations diverge. The complaint is not that the authors used a Monte Carlo estimate; any such study must. It is that the estimate's precision is the load-bearing part and it is undocumented. I do not think this warrants rejection: the qualitative findings about nouns/punctuation and distinguishability may well survive a larger n, and the authors already show several internal baselines. The reader's CONDITIONAL verdict is exactly right; no change. If the test reveals instability, the condition should be upgraded to a requirement that all headline analyses carry confidence intervals or a larger-n replication.","tokens_in":13201,"tokens_out":5002,"duration_ms":63055,"concrete_test":"Rerun the optimization/influence pipeline for word-stories and Pythia-1.4B with n = 64 and n = 256 (or, if the released code already sets n, multiply it by 4), using an independently sampled continuation set for evaluation. Then compute bootstrap confidence intervals for the influence scores s_i in Eq. 2 from resampled d_i. If the set of prompts passing dKL <= 10.0 changes materially, or the top-3 influence ranks in Table 1/Figure 2 overlap across bootstrap replicates, the headline composition and probe findings are not stable; if they are stable, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 defines functional similarity through Eq. 1: dKL(p*||p) = (1/n) Σ [log P(d_i|p*) - log P(d_i|p)] with d_i ~ P(·|p*). This single estimator does triple duty: Appendix A.2 uses dKL <= 10.0 to decide which prompts count as 'optimized'; Eq. 2 turns it into token influence scores and produces the rank-1 'outsized effect' in Figure 1; Section 6.3 computes layer-wise dKL curves. The manuscript never states n, gives no error bars, and does not check whether the dKL <= 10.0 filter is stable. With finite n, the estimator is unbiased but high-variance and can even be negative; GCG can minimize the empirical average by fitting the particular sampled continuations. If n is small, prompts that pass the filter need not be functionally similar on fresh continuations, so the token-composition (Figure 2), rarity (Figures 3-4), probing (Figure 5), and layer-wise KL (Figure 7) results describe a collection selected partly by sampling noise. The 'dKL = 0 iff equivalent' statement is only true in the n -> infinity limit. This is not an external-validity objection; it is an internal statistical robustness check on the quantity that defines the object of study.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates the structure and internal processing of machine-generated (\"optimized\") prompts obtained by running Greedy Coordinate Gradient (GCG) under the \"evil twins\" KL objective (Melamed et al., 2024). The authors define functional similarity via an empirical KL divergence (Eq. 1), use it to filter optimized prompts (dKL <= 10.0, Appendix A.2), and derive token influence scores (Eq. 2) that rank tokens by the effect of their removal. Across a custom word-level Tiny Stories model and 18 open models, they report that (i) the top-ranked token has an outsized influence, (ii) influential tokens are predominantly punctuation and nouns, (iii) optimized prompts contain tokens that are rarer in the pre-training corpus than natural language counterparts and deviate from Zipfian distributions, (iv) sparse linear probes trained on intermediate activations distinguish optimized from natural prompts, and (v) instruction-tuned models exhibit similar layer-wise KL trajectories. The paper also presents causal-intervention experiments (feature ablation) and layer-wise KL divergence curves. The main claims are empirical and descriptive rather than theoretical, and the authors release code and models.","tokens_in":13487,"tokens_out":6713,"duration_ms":75586,"significance":"If the underlying measure is statistically sound, this is a valuable empirical contribution: it is among the first systematic studies of the token-level and representation-level properties of GCG-optimized prompts across multiple model families. The custom word-level Tiny Stories model is a good idea for sidestepping subword interpretability, and the breadth of models (18, including base and instruction-tuned variants) strengthens the descriptive findings. The sparse-probing and layer-wise KL analyses target a question of clear safety relevance: detecting and understanding adversarial or machine-generated prompts. However, the central similarity measure (Eq. 1) is an unvalidated finite-sample estimator, and several methodological details (sample size n, probe train/test separation, POS tagging of subword tokens) are missing. These issues must be resolved before the empirical claims can be fully trusted, but they are fixable within the manuscript's scope.","major_comments":[{"comment":"The sample size n used in the empirical KL divergence (Eq. 1) is never reported. This estimator is load-bearing in three places: Appendix A.2 uses dKL <= 10.0 to decide which prompts count as 'optimized'; Eq. (2) converts it into token influence scores, producing the rank-1 'outsized effect' in Figure 1; and Section 6.3 computes layer-wise dKL curves. Because GCG optimizes this finite-sample objective, a small n would let the optimizer fit the particular sampled continuations, and the dKL <= 10.0 filter could admit prompts that are not functionally similar on fresh continuations. The statement that dKL(p*||p) = 0 if and only if the two prompts are functionally equivalent is only true in the n -> infinity limit. Please report n for every model, provide bootstrap confidence intervals or a fresh-continuation validation of the filter, and check the stability of the dKL <= 10.0 threshold.","section":"Eq. (1) and Appendix A.2"},{"comment":"The sparse probing experiments (Figures 5 and 10) do not describe any held-out split or cross-validation. Training a logistic regression classifier and evaluating it on the same optimized/natural prompt pairs can yield near-perfect accuracy from memorization, so the claim that optimized prompts are 'clearly distinguishable' (abstract and Section 6.1) needs generalization evidence. Please specify how prompts were split into training and test sets (or whether cross-validation was used), report the number of examples per class, and give accuracy with error bars across folds or random seeds.","section":"Section 6.1 / Appendix A.3"},{"comment":"The part-of-speech tagging procedure is underspecified for BPE-based models. Table 1 lists tokens such as ':**' and 'OKnote?).' as single tokens; it is not clear how spaCy assigns a POS tag to such strings, or whether subword tokens are first merged into whole words. Because the claim that influential tokens are 'primarily nouns and punctuation' is central, please describe the exact tagging pipeline, including how subword boundaries are handled, and consider reporting a sanity check on the word-level Tiny Stories model separately.","section":"Section 4.1 / Figure 2"}],"minor_comments":[{"comment":"The claims that optimized prompts have a 'visibly' different distribution and rely on rarer tokens would be strengthened by a goodness-of-fit test (e.g., against a power law) and a two-sample test (e.g., Kolmogorov-Smirnov) rather than visual inspection alone.","section":"Figures 3 and 4"},{"comment":"Please report the regularization strength, feature normalization, and whether the MMD feature selection (Eq. 4) is computed on the training split only; otherwise there is a risk of leakage in the probe evaluation.","section":"Appendix A.3"},{"comment":"No error bars or confidence intervals are reported on the influence scores, probe accuracies, or KL curves; standard errors over prompts would substantially help assess the consistency of the effects, especially for models with few optimized prompts (e.g., llama3.1-8b-base has only 106 prompts after filtering in Table 2).","section":"Throughout"},{"comment":"The claim that instruction-tuned models 'follow a similar path' is based on qualitative inspection; a quantitative similarity measure (e.g., correlation or distance between layer-wise dKL profiles across model families) would make the claim falsifiable.","section":"Section 6.3 / Figure 7"},{"comment":"There are a few typos and wording issues: 'Sectio 6.1' in Section 6.3, 'propogating' in the related work section, and 'model suits' in the Figure 10 caption should read 'model suites'.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The primary barrier is the unvalidated finite-sample KL estimator: the manuscript does not report n, and this measure defines the object of study (the filtered set of 'optimized' prompts), the influence scores, and the layer-wise analyses. If the authors can supply n, demonstrate stability of the dKL <= 10.0 filter (e.g., via fresh-continuation evaluation), and add the missing probe train/test details, the paper's central descriptive claims are plausible and likely acceptable. I do not see citation or novelty-disclosure concerns beyond a heavy reliance on the authors' own prior framework (Melamed et al., 2024), which is cited appropriately. The scope fits an NLP/interpretability venue well."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a solid descriptive study of GCG-optimized prompts, and it does more than its predecessors. The systematic sweep across 18 models, the word-level tokenizer for Tiny Stories, the corpus-rare token analysis, and the sparse probing of activations are all new relative to the cited prior work on filler tokens and OPT attention. The finding that optimized prompts lean on punctuation and nouns is real, though their own appendix shows natural language prompts lean the same way, so the \"primarily consist of\" wording should be softened. The layer-wise KL paths for instruction-tuned models are interesting and worth reporting even if they are purely descriptive.\n\nThe elephant in the room is Equation 1. The paper never reports n, the number of sampled continuations, and this estimator does triple duty: it defines \"optimized\" via the dKL <= 10 filter, it produces every influence score, and it generates the layer-wise KL curves. With finite n the estimator is high-variance and GCG can overfit to the sampled continuations. The stress-test concern is legitimate: if n is small, some prompts in the filtered set could be selected partly by sampling noise. That does not sink the paper. The qualitative findings — rare tokens, high probe accuracy, the shape of the layer-wise curves — are unlikely to flip entirely, and the authors do provide code and data, which lets a skeptical reader check. But they need to report n, give error bars on the influence and KL curves, and ideally show that the dKL<=10 filter is stable under different continuation samples. These are standard robustness checks, not fatal flaws.\n\nTwo smaller issues: the probe classifiers are described without an explicit held-out split, so I cannot tell whether the reported accuracy is in-sample or out-of-sample; and POS tagging of BPE subword tokens for the open models is not explained, which matters for the grammatical-category claim. Both are fixable in revision.\n\nThe citation pattern looks fair. The paper builds on the authors' own evil-twins framework, but it cites that work transparently and the independent findings (rare tokens, probe separability) do not stand or fall on the KL formulation alone.\n\nWho is this for? People working on LLM safety, jailbreak detection, and interpretability. It is not a breakthrough, and it does not open a new capability, but it gives the field a clearer empirical picture and a plausible detection signal. It deserves a serious referee. I would send it out with a request for the statistical details above, and I would probably cite the rare-token result if I worked in that area.","headline":"Useful empirical map of GCG-optimized prompts with genuinely new findings, but the load-bearing KL estimator needs error bars and a stated sample size before publication.","tokens_in":14024,"tokens_out":1401,"would_cite":true,"duration_ms":19569,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Optimized prompts are not irreducible gibberish: across 18 models, they are made mostly of rare nouns and punctuation, and their activations are sparsely distinguishable from natural language.","keywords":["optimized prompts","discrete prompt optimization","Greedy Coordinate Gradient","adversarial prompts","jailbreak","activation interpretability","sparse probing","token frequency"],"falsifier":"Recompute Eq. 1 with a much larger and explicitly reported sample of continuations and re-run the token-influence and layer-wise KL analyses: if the most influential tokens stop clustering in punctuation and nouns, or optimized tokens stop being rarer than natural-language tokens, the central claim fails.","tokens_in":13001,"feed_emoji":"🔤","tokens_out":6428,"duration_ms":73910,"temperature":0.7,"pith_summary":"This paper asks whether machine-generated \"optimized\" prompts, strings of tokens that look like gibberish but steer language models toward specific behaviors, are truly uninterpretable. It argues they are not: across a word-level model and 18 open models, optimized prompts are made largely of nouns and punctuation, and their tokens are rarer in the training data than those of natural-language prompts. Internally, sparse sets of activations distinguish optimized from natural prompts with high accuracy, even though the two prompt types produce functionally similar continuations. The paper also reports that instruction-tuned models follow a similar layer-by-layer path, with later layers pulling optimized and natural prompts back toward functional alignment. A sympathetic reader would care because this is evidence that adversarial-looking prompts carry discoverable structure rather than being irreducible noise.","feed_headline":"Garbled AI prompts are mostly rare nouns and punctuation","feed_subtitle":"Machine-generated prompts that look like gibberish leave a distinct, sparsely detectable trace in a model's activations.","key_machinery":"The central object is the prompt-pair functional-similarity measure $d_{KL}(p^*\\|p)$ (Eq. 1), an empirical KL divergence between the continuation distributions of a natural prompt and its optimized twin. Around it, the paper builds an influence score (Eq. 2) that removes each token and re-measures $d_{KL}$ to identify load-bearing tokens, and a mean-difference feature ranking (Eq. 4) that scores each hidden-state dimension by how differently it responds to optimized versus natural prompts. These quantities, plus the layer-wise KL divergence obtained by projecting each layer's last-token representation into vocabulary space, carry the argument: composition claims come from influence scores, and representational claims come from the probe and layer-wise KL analyses.","core_discovery":"The central claim is that optimized prompts have a consistent, analyzable structure: the tokens that matter most for behavior are predominantly nouns and punctuation, and the full token set skews toward tokens that are rare in the pretraining corpus. Inside the model, sparse subsets of activations separate optimized from natural prompts with high accuracy, while baseline comparisons between natural or random prompts are near chance. Layer-wise analysis shows the model rebuilding functional similarity mainly in the later layers, and instruction-tuned models follow a similar representational path across model families.","pith_inferences":["A control the paper does not run: retrain the word-level model on a corpus that also contains the optimized prompts' rare tokens; if their influence and detectability drop, the corpus-rare explanation is confirmed rather than merely associated.","The layer-wise convergence result suggests a defense that is not tested here: a training or decoding-time objective that penalizes divergence between natural and optimized hidden states in the last layers could reduce jailbreak success.","Because natural-language prompts also lean on punctuation and nouns (Appendix C), the pattern may reflect a general autoregressive tendency in all prompts, and a testable question is whether token-frequency rarity alone predicts influence score without the optimization procedure."],"forward_implications":["If optimized prompts are detectable from sparse activation subsets, a lightweight linear classifier on intermediate-layer activations can flag likely jailbreak prompts before the model finishes generating a response, which the paper itself suggests.","If the later layers are what restore functional similarity between optimized and natural prompts, then interventions in the final layers are the most promising place to disrupt or steer optimized-prompt behavior.","If optimized prompts rely on corpus-rare tokens, then tokenizer design and pretraining-data coverage shape how easy a model is to optimize against, so attack difficulty should vary with tokenization.","If instruction-tuned models share a similar representational path, the qualitative findings should transfer across model families rather than being unique to any single architecture."],"supporting_citations":[{"why":"Supplies Greedy Coordinate Gradient, the discrete optimization algorithm used to generate all optimized prompts.","marker":"Zou et al., 2023b"},{"why":"Defines the evil twins objective and the empirical KL divergence in Eq. 1 that every influence and layer-wise measurement builds on.","marker":"Melamed et al., 2024"},{"why":"Contributes the sparse probing and mean-difference feature scoring method used in Section 6.1.","marker":"Gurnee et al., 2023"},{"why":"Provides the Tiny Stories corpus used to train the word-level model and to create optimized prompts.","marker":"Eldan and Li, 2023"},{"why":"Concurrent finding that optimized prompts contain low-impact filler tokens and depend on the final token, which the influence analysis engages with directly.","marker":"Rakotonirina et al., 2024"}],"fun_headline_variants":["Rare nouns and punctuation make up AI prompt gibberish","AI prompt gibberish deciphered: rare nouns and punctuation","The rare-token secret behind AI prompt gibberish","AI prompt gibberish: a sparsely detectable trace"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole analysis depends on the assumption that the empirical KL divergence in Eq. 1, estimated from a finite set of sampled continuations, accurately measures how functionally similar two prompts are, and the paper does not report how many continuations were sampled.","fun_headline_variants_meta":{"raw":{"variants":["Rare nouns and punctuation make up AI prompt gibberish","AI prompt gibberish deciphered: rare nouns and punctuation","The rare-token secret behind AI prompt gibberish","AI prompt gibberish: a sparsely detectable trace"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000579,"raw_usage":{"total_tokens":2634,"prompt_tokens":755,"completion_tokens":1879,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":371,"completion_tokens_details":{"reasoning_tokens":1809}},"tokens_in":371,"tokens_out":1879,"duration_ms":13245,"temperature":1.0,"reasoning_tokens":1809,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:56:23.110835+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute Eq. 1 with a much larger and explicitly reported sample of continuations and re-run the token-influence and layer-wise KL analyses: if the most influential tokens stop clustering in punctuation and nouns, or optimized tokens stop being rarer than natural-language tokens, the central claim fails.","supporting_citations":[{"cited_title":"Howie Huang, and Enric Boix-Adser \\`a","cited_arxiv_id":null,"evidence_quote":"Defines the evil twins objective and the empirical KL divergence in Eq. 1 that every influence and layer-wise measurement builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Concurrent finding that optimized prompts contain low-impact filler tokens and depend on the final token, which the influence analysis engages with directly."}],"review_version":1}