{"id":"8337b2f1-600a-4a95-afec-7336670bf411","arxiv_id":"2505.00626","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Fine-tuned LLMs separate roles via task-type and begin-of-text shortcuts; PFT, which inserts a gap into position IDs during fine-tuning, reduces those shortcuts and improves closed-domain attack robustness.","lead":"This paper finds that fine-tuned language models often tell system and user messages apart using shortcuts, like recognizing the type of task or which text comes first, rather than truly understanding the role boundary. The authors show that renumbering token positions during fine-tuning makes role separation more robust and does not hurt normal task performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PFT's role-separation mechanism is never validated with the paper's own role-swap diagnostic; Table 2 only shows attack accuracy, so the central claim that the position-ID gap teaches role distinction remains unproven.","rationale":"The paper's central contribution is that PFT teaches models to separate roles by reinforcing invariant role-boundary signals. The most direct evidence would be a demonstration that role information controls model behavior. The role-swap test in Section 4.1 is exactly such a demonstration and was used to expose the task-type shortcut in the initially fine-tuned model. The paper never reuses this diagnostic on PFT models; instead, Table 2 and Figures 5/7 report attack accuracy and insertion robustness. Attack accuracy can improve for many reasons—e.g., a bias toward 'Access Denied' or a general distrust of later tokens—without implying that the model distinguishes roles. The inserted-text results are also ambiguous: since the gap is always placed between system and user, the model could be learning a positional template ('anything after the gap is data') rather than the role concept. Running the role-swap diagnostic settles this: if PFT models change their outputs appropriately after swapping system and user contents, the gap plays a causal role; if not, the headline claim is overstated. The reader's CONDITIONAL verdict is appropriate, and the requested check should be a condition for acceptance. This is a missing verification step, not a claimed falsification.","tokens_in":11939,"tokens_out":12075,"duration_ms":126642,"concrete_test":"Run the Section 4.1 role-swap diagnostic on the PFT-trained and vanilla SFT models: for a held-out set of benign prompts, create pairs in which the system and user contents are swapped while keeping role delimiters and the PFT gap d fixed. Measure (i) whether the greedy output changes and (ii) whether it follows the new system instruction. If PFT models still produce near-identical outputs after the swap, the position-ID gap is not teaching role distinction and the Table 2 gains likely come from another cue; if outputs track the new system role, the mechanism claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 4.1, the authors diagnose the task-type shortcut with a role-swap test: swap the textual contents of the system and user roles while keeping the structure fixed, and check whether the model's output changes. A model with true role separation should follow the new system instruction; a model relying on task-type association keeps producing the old output. This diagnostic is never applied to the PFT models. Table 2 reports accuracy on Gandalf/TensorTrust attacks, but those gains could arise for reasons unrelated to role grounding—for example, PFT might make the model more conservative (biased toward 'Access Denied') or make it adapt to the specific position distribution (gap d=512/256) rather than to the role boundary. The inserted-text stress tests in Figure 5 are also consistent with a positional template ('tokens after the gap are data') rather than a genuine role concept learned from delimiters. Without the role-swap check, the abstract's claim that 'manipulating position IDs helps the model learn clearer distinctions' is an interpretation, not a demonstrated result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether fine-tuned LLMs genuinely learn to separate system and user roles in multi-role prompts, as opposed to relying on shortcuts. It proposes a controlled closed-domain framework where training only uses benign data while evaluation uses adversarial prompts, and uses this framework to identify two shortcuts: task-type association and proximity to begin-of-text. The paper then introduces Position-enhanced Fine-tuning (PFT), which modifies position IDs to create a gap between system and user tokens, and presents experiments on Llama-3-8B-Instruct and Gemma-2-9b-it showing that PFT improves robustness on several adversarial attack datasets and on inserted-sentence stress tests, while maintaining ordinary-task performance as measured by proxy metrics.","tokens_in":12059,"tokens_out":5729,"duration_ms":55505,"significance":"The paper makes a valuable conceptual contribution by distinguishing true role separation from pattern memorization, and by proposing a simple training-time intervention (PFT) that appears to improve adversarial robustness in closed-domain settings. The controlled framework is a useful methodological step for the prompt-injection community. However, the evidence for the central mechanism claim is incomplete: the role-swap diagnostic used to identify the task-type shortcut is never applied to PFT, and the experimental protocol lacks multiple seeds or error bars. The finding that PFT regresses on Gandalf Summarization for Llama also complicates the 'for free' claim. If the missing diagnostics are added and the statistical uncertainties are addressed, the paper would be a solid contribution.","major_comments":[{"comment":"The role-swap diagnostic introduced in Section 4.1 is never applied to the PFT models. Table 2 only reports attack accuracy, which could improve for reasons unrelated to role grounding, such as a conservative response bias or a positional template. Without showing that PFT changes outputs when the contents of system and user roles are swapped, the abstract's claim that PFT 'helps the model learn clearer distinctions' is an interpretation rather than a demonstrated result. Please apply the role-swap test to PFT models and report the outcome.","section":"Section 4.1 / Table 2"},{"comment":"PFT on Llama Gandalf Summarization achieves 85% accuracy, which is worse than SFT's 90%. The paper later claims that PFT improves robustness 'for free', but this regression on a core attack dataset is not discussed. This is relevant to the central claim of consistent improvement. The authors should either investigate and explain this failure or temper the 'for free' claim.","section":"Section 6.2 / Table 2"},{"comment":"All experiments appear to be single runs with no multiple seeds or error bars. Given the evaluation set sizes (114 samples for Gandalf Summarization, 500 for the other datasets), several reported differences are within sampling noise, e.g., TensorTrust Hijacking on Llama at 33% vs 37%. To support the claim that PFT 'consistently outperforms' the baselines, please report means and standard deviations over at least three seeds, or provide confidence intervals.","section":"Appendix B"},{"comment":"The utility evaluation uses log-likelihood under the base model and KL divergence on Alpaca prompts as proxies for generation quality. These metrics measure deviation from the base model, not task performance. The claim that PFT 'does not compromise performance on ordinary data' is too strong when only proxy metrics are reported. Please include a direct instruction-following benchmark (e.g., AlpacaEval or a small human-evaluated set) or rephrase the claim to reflect the proxy nature of the measurement.","section":"Section 6.2 / Appendix B"}],"minor_comments":[{"comment":"Typo: 'differntiates' should be 'differentiates'.","section":"Section 3"},{"comment":"Typo: 'privildged' should be 'privileged'.","section":"Section 4.1"},{"comment":"Typo: 'exisitng' should be 'existing'.","section":"Section 7"},{"comment":"The x-axis label 'Num Shifted T okens' has an unintended space; it should be 'Num Shifted Tokens'.","section":"Figure 3"},{"comment":"The position-ID diagram would benefit from an explicit annotation of the gap size d, as the current notation 'd+9' etc. is not immediately clear without reading the text.","section":"Figure 4"},{"comment":"The paper states that model selection for PFT uses validation loss on benign data. It would be helpful to clarify whether the adversarial evaluation sets were used for any hyperparameter selection; if not, state this explicitly, since the closed-domain claim depends on it.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper has a sound core idea and a reusable evaluation framework, but the missing role-swap test for PFT is a significant gap that the authors can likely address with existing infrastructure. The Gandalf Summarization regression and the single-run protocol also need attention. I would not reject the paper, as the framework and the PFT idea are interesting, but the evidence currently does not fully support the central mechanistic claim. A major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2505.00626. First, the controlled framework for studying role-separation learning is genuinely useful: training only on benign data and evaluating on adversarial data separates pattern memorization from actual role grounding, and the two shortcuts they identify (task-type association and proximity to begin-of-text) are clearly demonstrated. Second, the paper's main intervention, Position-enhanced fine-tuning (PFT), is plausible but under-validated: the authors never apply their own role-swap diagnostic to the PFT models. So when the abstract claims PFT teaches 'clearer distinctions,' that is an interpretation, not a demonstrated result. The attack-accuracy gains in Table 2 could come from the model becoming more conservative or from adapting to a specific positional template rather than from genuine role grounding.\n\nThe paper does several things well. The controlled framework is a real contribution to the prompt-injection evaluation literature, and the demonstration of the begin-of-text proximity shortcut via null-token and position-shift controls is clean. PFT is simple and the utility checks (password accuracy, Alpaca log-likelihood, KL divergence) show that the manipulation does not obviously hurt ordinary performance. Running on both Llama-3 and Gemma-2 strengthens the empirical case.\n\nThe soft spots are real but mostly fixable. No error bars or multiple seeds; the protocol seems to be a single run. There is a regression on Gandalf Summarization for Llama (85% vs 90% for SFT), which weakens the claim that PFT is uniformly better. The closest concurrent work (Wu et al., 2024) is cited but never compared. No code or data is provided, which limits independent verification. But the most important gap is the missing role-swap test: it is the paper's own diagnostic for whether the model follows the system role, and it is absent from the PFT evaluation. Adding that single experiment would go a long way.\n\nWho is this for? Researchers working on prompt-injection defenses and multi-role LLM safety. They will find the framework and shortcut analysis valuable even if PFT itself needs more evidence. The paper deserves a serious referee. I'd send it to review, but I'd ask for the role-swap test on PFT, multi-seed results, and a head-to-head with Wu et al. If those come back clean, it's a solid contribution.","headline":"A valuable controlled framework and a plausible but under-validated intervention: the role-swap diagnostic never reaches the PFT models, so the mechanism claim outruns the evidence.","tokens_in":12653,"tokens_out":3036,"would_cite":true,"duration_ms":28672,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned LLMs learn role separation through shortcuts, and a position-ID gap fixes it.","keywords":["role separation","prompt injection defense","shortcut learning","position IDs","fine-tuning","instruction hierarchy","LLM security","position-enhanced fine-tuning"],"falsifier":"Run a PFT model (e.g., Llama with $d=512$) on adversarial prompts where the system content is moved after the user content or where the gap is changed to a value not seen in training; if extraction or hijacking accuracy collapses, the model has adapted to the training position distribution rather than learning the gap as a role invariant.","tokens_in":11635,"feed_emoji":"🔐","tokens_out":6172,"duration_ms":50077,"temperature":0.7,"pith_summary":"The paper argues that fine-tuned LLMs usually do not learn to actually distinguish system and user roles; instead they latch onto two shortcuts: tasks that resemble their training data, and nearness to the beginning of the prompt. The authors show this by training on 'benign' data and testing on adversarial prompts, which stops the model from memorizing attack patterns. They propose Position-enhanced Fine-tuning (PFT), which modifies position IDs to create a fixed gap between system and user tokens, and report that it improves resistance to extraction and hijacking attacks on two models without hurting ordinary performance. The pith is that role boundaries need a dedicated, token-level invariant signal rather than repeated data patching.","feed_headline":"Position-ID gap teaches LLMs to separate roles for real","feed_subtitle":"A training tweak stops models from using prompt position as a shortcut, improving resistance to hijacking and extraction attacks.","key_machinery":"The central object is Position-enhanced Fine-tuning (PFT), a training-time manipulation of position IDs. During supervised fine-tuning, each token keeps its original order inside the system and user sections, but a fixed gap $d$ is inserted between the last system token and the first user token, giving every system token a numerically smaller position than every user token. This turns role identity into a persistent, token-wise numeric signal that the model can learn, in contrast to delimiter tokens or data augmentation, which the paper argues only patch individual shortcuts.","core_discovery":"The central claim is that standard supervised fine-tuning on multi-role prompts produces only an illusion of role separation: models pass adversarial evaluations by exploiting task-type association and proximity to begin-of-text, not by genuinely marking system tokens as privileged. The paper demonstrates that when non-essential instructions are inserted before the key system instruction, fine-tuned models lose the ability to ignore user instructions, even though they perform well on the original evaluation split. The proposed fix, Position-enhanced Fine-tuning (PFT), manipulates position IDs so that the first user token is assigned position $k+1+d$ when the system section ends at position $k$, creating a clear numerical boundary while preserving internal order within each role. The paper reports that PFT alleviates both shortcuts on Llama-3-8B and Gemma-2-9b across Gandalf and TensorTrust attacks, and that utility measured by password accuracy and Alpaca log-likelihood is unchanged relative to standard SFT.","pith_inferences":["If PFT works because the gap creates a learnable scalar boundary, other token-wise signals (e.g., role-specific biases added to hidden states or a dedicated role token) might achieve similar gains with less dependence on a hyperparameter like $d$; the paper's own related-work discussion points toward role-specific embeddings.","The diagnosis implies that existing evaluations may overstate safety: a model that refuses user-role overrides with high frequency is actually treating the override as an instruction regardless of which role contains it, so its behavior is not role-based at all.","A testable extension is to apply PFT to open-domain settings where user inputs sometimes contain harmless instructions; the paper excludes this setting, so whether the position gap helps or harms legitimate instruction-following in user input remains unknown.","The gap magnitude $d$ is selected by validation loss for each model; this leaves open the possibility that PFT's robustness is specific to the training distribution of positions, and a model trained with a variable or randomized gap might generalize better."],"forward_implications":["Fine-tuned role-separation models cannot be trusted to generalize to new prompt structures, because evaluations that place adversarial instructions only in the user role cannot distinguish role learning from pattern matching.","Data augmentation (symmetrization, inserting filler text) mitigates known shortcuts but invites a cycle of new shortcuts; a token-wise intervention is needed.","Position-enhanced fine-tuning improves resistance to extraction and hijacking attacks beyond standard SFT and augmentation, while preserving accuracy and generation quality.","The success of PFT suggests that LLMs can adapt to altered position IDs during fine-tuning without losing ordinary capabilities."],"supporting_citations":[{"why":"Supplies the instruction-hierarchy finetuning framework used in the Section 2 role-sensitivity test that motivates the paper's question.","marker":"Wallace et al., 2024"},{"why":"Provides the TensorTrust extraction and hijacking attack samples used for adversarial evaluation.","marker":"Toyer et al., 2023"},{"why":"Gandalf Ignore attack dataset used for evaluation.","marker":"Lakera AI, 2023a"},{"why":"Gandalf Summarization attack dataset used for evaluation.","marker":"Lakera AI, 2023b"},{"why":"Attention sink phenomenon used to explain why proximity to begin-of-text becomes a shortcut.","marker":"Xiao et al., 2023"},{"why":"LoRA, the parameter-efficient finetuning method used for all models.","marker":"Hu et al., 2021"},{"why":"Base model Llama-3-8B-Instruct for primary experiments.","marker":"AI@Meta, 2024"},{"why":"Gemma-2-9b-it, used to validate the findings on a second architecture.","marker":"Team, 2024"},{"why":"Alpaca prompts used to measure utility (log-likelihood and KL divergence) of finetuned models.","marker":"Taori et al., 2023"}],"fun_headline_variants":["Position-ID tweak fixes LLM role-separation illusion","Fine-tuning only fakes role separation—position IDs fix it","Position-enhanced fine-tuning: real role separation for LLMs","LLMs exploit role shortcuts; position-ID gap makes separation real","Role separation illusion: position-ID gap is the fix"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The one assumption everything rests on is that the fixed position gap between system and user tokens teaches a general rule for telling the two roles apart, rather than the model just memorizing the layout of the training prompts.","fun_headline_variants_meta":{"raw":{"variants":["Position-ID tweak fixes LLM role-separation illusion","Fine-tuning only fakes role separation—position IDs fix it","Position-enhanced fine-tuning: real role separation for LLMs","LLMs exploit role shortcuts; position-ID gap makes separation real","Role separation illusion: position-ID gap is the fix"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000269,"raw_usage":{"total_tokens":1644,"prompt_tokens":993,"completion_tokens":651,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":569}},"tokens_in":609,"tokens_out":651,"duration_ms":6174,"temperature":1.0,"reasoning_tokens":569,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:37:55.228515+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a PFT model (e.g., Llama with $d=512$) on adversarial prompts where the system content is moved after the user content or where the gap is changed to a value not seen in training; if extraction or hijacking accuracy collapses, the model has adapted to the training position distribution rather than learning the gap as a role invariant.","supporting_citations":[],"review_version":1}