{"id":"c28126fa-2b2b-4181-ab13-4e10bb443e3a","arxiv_id":"2412.15275","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Hidden 'high score' states in an LLM grader can be amplified by an optimized text suffix containing 'user', inflating grades and transferring across models, with a chat-template change reducing the effect.","lead":"This paper shows a way to trick AI essay graders into giving higher scores by appending a short nonsense phrase to an essay. The phrase is found by reading the AI's internal activity patterns, and it also works on commercial AI systems like Gemini.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Score gains are not yet causally attributed to the neural readout direction; missing random-target and random-suffix controls leave the 'neural activity guided' contribution underdetermined.","rationale":"The reader's weakest assumption concerned whether the readout direction is a universal cognitive state versus a dataset artifact. My concern is adjacent but more specific: even if the readout direction is universal, the paper does not show it is causally responsible for the attack. The optimized suffixes contain 'user' and positive-sentiment tokens, and the ablations show these tokens matter, but no experiment varies the optimization target while holding the search procedure fixed. Consequently, the observed score gains and cross-model transfer could be produced by ordinary GCG discovering a prompt-injection pattern, with the neural readout merely along for the ride. This is load-bearing because the paper's novelty is the neural-activity-guided method, not the existence of adversarial suffixes. I do not think the paper should be rejected: the empirical attack is plausible and the template fine-tuning experiment is a meaningful step. The right response is to keep the reader's CONDITIONAL verdict and add an explicit condition: demonstrate that the readout direction, not just the search procedure, drives efficacy. My agreement_with_reader is partial because they emphasize universality of the readout across problems/templates, while I emphasize the missing causal control separating the readout target from the optimizer's discovery of a trivial 'user' bias. The concrete test above would settle this by comparing readout-targeted optimization against random-target and dummy-target baselines; if the gains vanish without the readout direction, the central claim stands, and if they do not, the 'neural activity guided' framing needs substantial revision.","tokens_in":13809,"tokens_out":4759,"duration_ms":53189,"concrete_test":"Run the identical GCG pipeline (same 20-token ASCII search, 1000 iterations, 10 seeds, same cleanup) on the same two problems/templates, but replace the averaged readout row in the loss with (a) a random unit vector in R^4096, (b) a fixed dummy token, and (c) the low-score readout row. Evaluate all suffixes on the same held-out essays (problems #3/#4, templates #3/#4) with paired 95% confidence intervals, and report whether 'user' appears in each optimized suffix. If random-target and readout-target suffixes have statistically indistinguishable mean score gains, or if all optimized suffixes converge to 'user'-based injections, the readout direction is not doing the causal work attributed to it in Section 3.3.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central methodological claim is that maximizing the projection of the layer-16 final-token activation onto the averaged high-score readout row (Section 3.3) is what makes the suffixes effective. No control isolates this target direction. Table 1 shows every optimized suffix begins with the token 'user'; Figure 6 shows removing 'user' or '.user' destroys the gain; Appendix D shows flipping sentiment words reverses the effect. These observations are equally consistent with a simpler mechanism: GCG discovered a role-token injection ('user') plus positive-sentiment words, and the readout direction is causally inert. The ablation in Figure 6 removes tokens from suffixes already selected by the readout objective, so it cannot distinguish whether the objective mattered. Without (i) a random-ASCII-suffix baseline, (ii) a length-matched non-adversarial suffix control, or (iii) suffixes optimized against random or negative readout directions, the empirical score gains cannot be attributed to 'neural activity guided' optimization. Since this attribution is the paper's stated contribution (Section 1, contributions 2 and 3), the central claim is underdetermined as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage method for attacking LLM-based essay graders: first train a linear readout on internal activations of Llama3.1-8B-Instruct to identify a representation associated with high scores, then use the GCG algorithm to optimize an adversarial suffix that maximizes projection onto the averaged high-score readout direction. The authors report that the resulting suffixes increase LLM-assigned scores on held-out essay problems and prompt templates, transfer to several other open- and closed-source models including Gemini, and contain a recurring 'user' token whose removal sharply reduces effectiveness. They further attribute this 'nonsense user effect' to the chat template used in supervised fine-tuning and show that replacing 'user' with a special token in the template reduces the attack's effectiveness.","tokens_in":14057,"tokens_out":2269,"duration_ms":24370,"significance":"If the findings hold, the paper makes a useful contribution by demonstrating a systematic, optimization-based method to uncover and exploit hidden biases in LLM evaluators, with evidence of transfer across models. The held-out evaluation design (unseen essay problems and prompt templates), the cross-model transfer results, and the supervised fine-tuning experiment as a causal test of the 'user' bias are notable strengths. However, the central claim that the effectiveness is specifically due to the neural readout direction is not yet fully supported, because key control conditions are missing.","major_comments":[{"comment":"The manuscript does not include a random-suffix baseline or a control in which suffixes are optimized against a random or negative readout direction. The observed score gains could plausibly arise from the GCG-discovered token sequence itself (e.g., the 'user' role token plus positive sentiment words), independent of the readout direction. Since the paper's stated contribution is 'neural activity guided' adversarial prompting, the authors should add (i) a length-matched random ASCII suffix control, (ii) an identical GCG optimization against a random or negative target direction, and (iii) a non-adversarial but semantically similar suffix, and show that the readout-targeted suffixes significantly outperform these controls.","section":"Section 3.3, Table 1, Figure 5"},{"comment":"Transfer results are presented only as scatter plots without aggregate effect sizes, confidence intervals, or statistical comparisons. The claim that suffixes are 'effective' across models and the qualitative comparison that larger models are less vulnerable need numerical summaries (e.g., mean and median normalized score gain with confidence intervals, proportion of essays with positive gain) and, ideally, a test comparing gains across conditions.","section":"Figure 5, Figure 10"},{"comment":"The token ablation removes tokens from suffixes that were already selected by the readout-based objective. This establishes that the 'user' token is important for the chosen suffixes, but it does not establish that the readout direction was causally responsible for the suffix's effectiveness. The missing random-direction control (noted above) is therefore load-bearing for the paper's main methodological claim.","section":"Section 4.2, Figure 6"},{"comment":"The supervised fine-tuning mitigation experiment should report whether the modified chat template changes the model's baseline grading behavior, e.g., agreement with human scores on unmodified essays. Without this, the reduced attack effectiveness could partly reflect a general degradation of instruction following or grading quality, rather than a removal of the specific 'user' bias.","section":"Section 4.3, Figure 7"}],"minor_comments":[{"comment":"There are numerous typos, including 'suffice' for 'suffix', 'the the' in Section 4.3, 'An very interesting' in Limitations, 'acoording' in Prompt Template #2, and 'EV ALUATION' in Prompt Template #4. A careful proofread is needed.","section":"Throughout"},{"comment":"The paper alternates between 'magic word' and 'magical word'; please use one term consistently.","section":"Section 1 and 4.2"},{"comment":"The evaluation of suffix cleanup is described as using 'Llama3.1-8B-Instruct-Turbo' with a footnote about FP8 quantization, but no details are given about how the quantized model differs from the base model in score calibration. Please clarify.","section":"Appendix/Section 3.3"},{"comment":"The choice to average the four highest-score readout vectors is operationally clear, but the paper should state whether using the average, rather than any individual readout vector, was validated (e.g., by comparing suffix success rates).","section":"Section 3.2"},{"comment":"No code or artifact link is provided. Given the complexity of the optimization pipeline, releasing code would substantially improve reproducibility; at minimum, please state an availability plan.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an interesting and timely topic, and the empirical transfer results are suggestive. However, the main methodological attribution—that the readout direction, rather than the discovered token patterns, drives the attack—needs stronger controls before publication. The missing random-suffix/random-direction baselines are the key issue. If the authors can add those controls and report aggregate statistics, the contribution would be solid for a security-oriented venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this one. The headline result—small adversarial suffixes inflate LLM essay grades and transfer to other models including Gemini—is real, and the paper's best contribution is the discovery of the 'user' token bias and its origin in SFT chat templates. The supervised fine-tuning experiment, where swapping 'user' for a special token kills most of the attack, is a clean causal test and the strongest part of the paper. That alone justifies a serious look.\n\nWhat is actually new: the observation that all optimized suffixes begin with 'user', the ablation study on Gemini showing that token is load-bearing, and the template-mitigation result. The readout-guided optimization is a natural extension of prior representation-engineering work (Zou et al., representation engineering; Li et al.; Xu et al.), and the transfer results across held-out problems, templates, and models are convincing at a qualitative level.\n\nNow the soft spots, in proportion. The central methodological claim—that maximizing projection onto the averaged high-score readout vector is what makes the suffixes work—is not isolated. There is no random-suffix baseline, no length-matched non-adversarial control, and no optimization against a random or negative target direction. The ablation in Figure 6 removes tokens from suffixes already selected by the readout objective, so it cannot tell us whether the objective mattered. The stress-test note is right: the results are equally consistent with GCG discovering the 'user' token plus positive sentiment words, with the readout vector being inert. That is a real gap, but it is not fatal to the empirical attack claims. The paper does not overclaim the mechanism as proven; it says 'we hypothesize' for the user effect. Still, the title and contributions 2 and 3 imply the neural readout is doing the work, and that is underdetermined.\n\nOther, smaller issues: no code or legible suffix strings are shipped, so independent replication is hard. There are no aggregate error bars or confidence intervals on the main transfer figures. The mitigation test only evaluates suffixes optimized on the original template, so it does not show that the template fix resists re-optimization. These are fixable in revision.\n\nWho this is for: people working on LLM security, prompt injection, and automated assessment. The empirical findings will be cited, and the template fix is a concrete defensive idea worth testing further. It deserves a serious referee, not a desk rejection. I would send it to peer review and ask the authors to add the missing control experiments and release the suffixes and code.","headline":"A genuinely useful empirical attack paper whose 'neural activity guided' contribution is under-supported, but the 'user' token story and the SFT template fix are real and worth refereeing.","tokens_in":14581,"tokens_out":1463,"would_cite":true,"duration_ms":16626,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tiny adversarial suffixes can inflate LLM essay grades by amplifying an internal 'high grade' state, and the trick traces to the word 'user' in chat templates.","keywords":["adversarial prompting","LLM grading","prompt injection","neural representations","cognitive state","chat template bias","automated essay scoring","GCG"],"falsifier":"Train the same linear readout on scores that have been randomly permuted across essays; if the resulting 'high-score' weight vectors still align across tasks as strongly as reported in Figure 3, the alignment is readout geometry rather than a meaningful cognitive state, and the attack should not transfer to new tasks.","tokens_in":13562,"feed_emoji":"🧠","tokens_out":5914,"duration_ms":51847,"temperature":0.7,"pith_summary":"This paper claims that a small, optimized string of tokens appended to an essay can make large language model graders award far higher scores than the essay deserves. The authors first train a linear readout that predicts the model's score from activations in a middle layer, then amplify the activation pattern associated with the highest score using gradient-based suffix optimization. They show the resulting suffixes boost scores on essay problems and prompt templates never seen during optimization, and that the attack transfers to other open-source models and to closed-source Gemini. The paper also identifies a 'magic word' — 'user' — that appears in every successful suffix, traces its power to the role-header token in standard supervised fine-tuning chat templates, and shows that replacing that token with a unique special token sharply reduces the attack's effectiveness.","feed_headline":"Tiny 'user' suffix tricks LLM graders into inflating scores","feed_subtitle":"The attack amplifies an internal 'high grade' state and traces it to a token in standard chat templates.","key_machinery":"The machinery is a linear readout plus an activation-amplification objective. A linear map $f(x;W,b)=Wx+b$ takes the residual-stream activation at layer 16, final token position, to logits over possible scores; the row of $W$ for the highest score is treated as a vector in activation space, and averaging those rows across tasks yields the target 'cognitive state.' The adversarial loss is the negative inner product between the current final-token activation and that target vector; the greedy coordinate gradient (GCG) algorithm, with token updates restricted to ASCII tokens, minimizes it by backpropagation, and a token-pruning step removes redundant suffix tokens. The readout both reveals the early grade judgment and supplies the direction that the attack amplifies.","core_discovery":"On the paper's own terms, the central discovery is that an LLM's grading decision is controlled by a specific direction in its internal activation space, and that direction can be hijacked. For Llama3.1-8B-Instruct, linear readouts at layer 16 predict the final score distribution from the activation at the last input token with correlation 0.8–0.9, which the authors interpret as the model forming its grade before producing any analysis. Averaging the readout rows for the highest score across two essay problems and two prompt templates gives a single target vector, interpreted as a cognitive state for 'excellent essay.' Optimizing a 20-token ASCII suffix to maximize the projection of the final-token activation onto that vector yields suffixes that raise scores on held-out essays, templates, and models; ablations show the token 'user' is essential, and supervised fine-tuning with a modified chat template that replaces 'user' with a unique special token drastically lowers the score gain.","pith_inferences":["Beyond the paper, if the high-score readout direction generalizes to other evaluation domains such as resume screening or peer-review-style judgment, the same suffix-optimization recipe could be ported with minimal re-optimization.","Beyond the paper, the 'user' effect suggests that other role-header tokens in other chat templates may create analogous attack surfaces, and measuring role-token activation directions could become a general audit method.","A concrete extension would be to measure the high-score readout direction in models fine-tuned with the modified chat template; if the direction remains aligned with high scores, the template fix removes this particular attack but not the underlying representation.","Since the attack is optimized white-box but transfers to black-box models, one could try to estimate the target activation direction from public checkpoints of similar models, potentially enabling black-box-only attacks not tied to the specific 'user' token."],"forward_implications":["The optimized suffixes improve scores on essay problem sets and prompt templates not used in optimization, so the attack does not require knowing the target prompt format in advance.","Suffix #1 raises scores on open-source models of various sizes and on Gemini-1.5-flash/pro, indicating that the vulnerability is shared across model families.","The final score is largely determined when the model finishes reading the essay, so the requested analysis text can be generated after the decision is already made.","Removing the token 'user' from a suffix significantly reduces its score gain, and replacing 'user' with a unique special token in the SFT chat template suppresses the attack.","Replacing positive words in the suffixes with antonyms lowers scores, showing the same mechanism can manipulate grades downward as well as upward."],"supporting_citations":[{"why":"Supplies the automated essay scoring dataset with human-graded scores, the ground truth and testbed for all experiments.","marker":"Hamner et al., 2012"},{"why":"Provides Llama3.1-8B-Instruct, the model whose internal activations are read out and attacked.","marker":"Dubey et al., 2024"},{"why":"Supplies the GCG algorithm used to optimize the adversarial suffixes.","marker":"Zou et al., 2023b"},{"why":"Introduces the nonsense math effect, the human-bias analogue the paper's method systematically reproduces.","marker":"Eriksson, 2012"},{"why":"Demonstrates that linear readouts of hidden states can reveal sentiment-related cognitive states in recurrent networks, the methodological precedent for the readout approach.","marker":"Maheswaranathan et al., 2019"},{"why":"Shows how to use internal representations to guide adversarial prompt optimization, a related approach the paper adapts to grading bias.","marker":"Li et al., 2024"},{"why":"Provides the RLHFlow SFT dataset used to fine-tune models with original and modified chat templates in the mitigation experiment.","marker":"Dong et al., 2024"}],"fun_headline_variants":["Neural attack flips LLM grading via hidden 'user' token","LLM graders fooled by activating hidden excellent state","Tiny suffix hijacks LLM's internal excellent state","user token guides adversarial prompts to inflate LLM grades","Adversarial prompts exploit chat template user bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the averaged high-score readout vector extracted from two essay problems and two prompt templates is a universal 'high grade' cognitive state, so maximizing its projection at the last token will raise scores on other essays, templates, and models.","fun_headline_variants_meta":{"raw":{"variants":["Neural attack flips LLM grading via hidden 'user' token","LLM graders fooled by activating hidden excellent state","Tiny suffix hijacks LLM's internal excellent state","user token guides adversarial prompts to inflate LLM grades","Adversarial prompts exploit chat template user bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000575,"raw_usage":{"total_tokens":2715,"prompt_tokens":948,"completion_tokens":1767,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":1696}},"tokens_in":564,"tokens_out":1767,"duration_ms":14457,"temperature":1.0,"reasoning_tokens":1696,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:17:23.880662+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same linear readout on scores that have been randomly permuted across essays; if the resulting 'high-score' weight vectors still align across tasks as strongly as reported in Figure 3, the alignment is readout geometry rather than a meaningful cognitive state, and the attack should not transfer to new tasks.","supporting_citations":[],"review_version":1}