{"id":"defcb879-d67e-44c7-9fc6-9988381093d1","arxiv_id":"2506.18053","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Token-permutation obfuscation preserves GPT-2-small's performance on an indirect-object task but makes head-level logit attribution and prompt reconstruction noisier, according to a from-scratch experiment with an uncontrolled baseline.","lead":"Scrambling the word labels a language model reads can keep a model working while making fine-grained analysis of its inner decisions much noisier, according to this study. The result matters for privacy-preserving AI because it suggests lightweight obfuscation may hide how models process individual prompts, though the paper's comparison is not fully controlled.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The token-permutation obfuscation is applied before training, so it is absorbed into learned embeddings and cannot alter internal activations; the observed interpretability differences are confounded with training-from-scratch versus a pretrained baseline.","rationale":"The reader's weakest assumption correctly identifies the missing-baseline confound: the obfuscated model is trained from scratch on Fineweb-Edu while the baseline is released pretrained GPT-2-small. My review goes further and argues that the specific obfuscation used—a tokenizer-level permutation applied before training—is theoretically incapable of altering internal activations when the model is trained from scratch, because the permutation is absorbed into the learned embedding and unembedding matrices. The resulting model is a reparameterization of a standard transformer with relabeled vocabulary; any observed differences in mechanistic interpretability must stem from training variation, not from obfuscation. This makes the central claim not merely under-supported but internally inconsistent: Section IV-B reports no head-level differences, while the abstract claims dramatic alterations. The paper is clearly written and shares code, which is commendable, but the experimental design cannot separate the effect of the permutation from the effect of training from scratch on different data. A matched-baseline experiment or a post-hoc permutation test would settle the matter, but absent such control the rejection of the central claim is warranted.","tokens_in":14748,"tokens_out":6139,"duration_ms":61186,"concrete_test":"Train GPT-2-small from scratch on the same Fineweb-Edu subset with the same seed, initialization, and training schedule in two conditions: (A) standard tokenizer, (B) the paper's permuted tokenizer. Then run the paper's logit-lens, residual-stream patching, and head-patching analyses on both models. If condition B's metrics match condition A up to a fixed token relabeling, the permutation is a no-op and the abstract's claim of dramatically altered activation patterns is refuted. Alternatively, apply the token permutation post-hoc to the released GPT-2-small weights by permuting embedding rows and unembedding columns; because the forward pass is identical, any difference in interpretability outputs would falsify the claim that this obfuscation method produces such differences.","verdict_should_be":"REJECT","load_bearing_attack":"The treatment in this paper is not an architectural obfuscation but a fixed permutation of token IDs in the tokenizer (Section III-A). When the model is trained from scratch with this tokenizer, the permutation is fully absorbed into the learned embedding and unembedding matrices: for any original token t, the embedding vector is row P(t) of the permuted embedding matrix, and the unembedding column is column P(t). The forward pass for a given text is therefore identical to that of a standard transformer with relabeled vocabulary entries; hidden activations, attention patterns, and residual-stream directions are unchanged in expectation. There is no mechanism by which this permutation can 'dramatically alter activation patterns within attention heads' unless the model is trained differently. The experiment compares a from-scratch model trained on Fineweb-Edu against the released pretrained GPT-2-small (Section III-F), simultaneously varying training data, initialization, random seed, and compute. The reported interpretability differences (diffuse patching signals, inverted head roles in Figures 5/7/9) are therefore attributable to training differences, not to the permutation itself. Moreover, Section IV-B states that 'no difference can be seen between the two models in terms of overall behavior of attention heads and layers,' directly contradicting the abstract's claim of dramatic alteration. Without a matched baseline (same data, same initialization, same schedule, only the permutation toggled), the central causal claim is untestable from the presented evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims to quantify how a tokenizer-level permutation obfuscation affects mechanistic interpretability in GPT-2-small. The authors train a 124M-parameter model from scratch on Fineweb-Edu with a seeded permuted tokenizer (Section III-A, III-B), then compare logit-lens attribution, residual-stream patching, component patching, and head patching with the released pretrained GPT-2-small on an indirect-object identification task (Section IV). They conclude that obfuscation preserves task performance and the layer-wise computational graph while making activation patterns more diffuse, inverting attention roles, and making prompt reconstruction less precise.","tokens_in":15016,"tokens_out":4980,"duration_ms":52364,"significance":"The question is relevant to both privacy-preserving inference and interpretability, and the paper has genuine strengths: it applies established interpretability methods to a concrete circuit task, targets a specific and falsifiable research question, and releases source code. If the comparison were controlled, the findings would be useful to the cs.CR and interpretability communities. However, the paper's central claim is not supported as written: the obfuscated and baseline models differ in training data, initialization, schedule, and compute, and the fixed pre-training token permutation is absorbed into learned embeddings. The reported interpretability differences therefore cannot be attributed to obfuscation. The internal contradiction between Section IV-B and the abstract further undercuts the main claim.","major_comments":[{"comment":"The comparison is confounded. The baseline is the released pretrained GPT-2-small, as indicated by the HellaSwag score of 0.2955 in Section III-F, while the obfuscated model is trained from scratch on Fineweb-Edu with a different optimizer setup, batch size, learning-rate schedule, and hardware configuration. Thus training data, initialization, random seeds, and compute all vary simultaneously. Any observed difference in patching or attribution patterns could be caused by these training differences rather than by the obfuscation map. A matched from-scratch baseline trained on the same data with the standard tokenizer is necessary to support the paper's causal claims.","section":"Section III-B and III-F"},{"comment":"The fixed token-permutation obfuscation is applied before training, and the embedding and unembedding matrices are learned from scratch with tied weights. A permutation applied at the tokenizer level is a relabeling of vocabulary indices: the model simply indexes different rows and columns of the embedding and unembedding matrices. For a fixed text, the forward pass is identical to that of a standard transformer with a relabeled vocabulary, so the permutation by itself has no mechanism to alter attention-head activation patterns. The paper does not hold training fixed, so it cannot measure any causal effect of the permutation on internal representations.","section":"Section III-A and III-D"},{"comment":"Section IV-B states that 'no difference can be seen between the two models in terms of overall behavior of attention heads and layers,' which directly contradicts the abstract's claim that obfuscation 'dramatically alters activation patterns within attention heads.' The paper needs to define precisely what counts as 'overall behavior' versus 'neuron-level' differences and provide a criterion for when an alteration is dramatic. As written, the central finding is ambiguous and internally inconsistent.","section":"Section IV-B vs. Abstract"},{"comment":"The abstract and conclusion claim that obfuscation makes prompt reconstruction 'too noisy' or 'far less precise,' but the paper contains no reconstruction experiment and defines no reconstruction metric. The reported analyses are logit-difference attribution and activation patching; these do not measure the success of any user-prompt reconstruction attack. This is an extrapolation beyond the evidence presented and should be either removed or supported by an explicit reconstruction evaluation.","section":"Abstract and Section V"}],"minor_comments":[{"comment":"References [14] and [15] appear to be the same EMNLP 2024 paper listed with different bibliographic details; one duplicate should be removed.","section":"References"},{"comment":"The sentence 'These correspond to some of the name movers and negative name movers discussed in [2]' cites a broad generative-AI survey rather than the actual indirect-object-identification circuit paper [34]; this citation should be corrected.","section":"Section IV-B"},{"comment":"There are several typographical and formatting issues, including 'aplication' in Section IV-B, 'IEEe Access' in the references, and inconsistent use of commas and hyphenation; a careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The deficiencies are not fixable by a textual revision: the conclusion requires a new experimental design with a matched from-scratch baseline and, ideally, a post-training obfuscation applied to a fixed pretrained model. The current manuscript also overstates its privacy relevance, since it evaluates no attack or reconstruction metric. The topic is potentially viable, but this version is not."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Frankly, this one misses the mark. The question is worth asking: does architectural obfuscation break mechanistic interpretability tools? But the experiment doesn't ask it. The seeded tokenizer permutes token IDs before training; a fixed permutation is absorbed into the learned embedding and unembedding matrices. The forward pass is identical to a standard GPT-2 with a relabeled vocabulary. So there is no obfuscation acting on hidden states or activations at inference time. What you're comparing is a from-scratch model trained on Fineweb-Edu against the released pretrained GPT-2-small. That is a training-difference confound, not a treatment effect.\n\nThe paper's own Section IV-B says 'no difference can be seen between the two models in terms of overall behavior of attention heads and layers,' which contradicts the abstract's claim that obfuscation dramatically alters activation patterns. That inconsistency alone would need explanation.\n\nCredit where due: the prose is readable, the code is public, and the tools (logit lens, residual/component/head patching) are applied carefully. The discussion of MLP0 and the negative heads is sensible. But the central causal claim is unsupported. The reconstruction difficulty is partly definitional—withholding the permutation map is a secret, not a discovered property.\n\nI don't think this deserves a formal peer-review slot as is. The fix is straightforward: use a real obfuscation (e.g., per-forward-pass secret permutation of hidden states, or a linear transform that is not absorbed into training), and train two matched models—same data, same init, same compute—differing only in the obfuscation. Then the interpretability differences could be attributed. Until then, the paper is a cautionary tale about confounds, but not a contribution to the privacy-interpretability literature. If a revised version does that, I'd be glad to referee it.","headline":"The paper asks a real question but the treatment isn't architectural obfuscation and the baseline isn't matched, so the central claim is untestable.","tokens_in":15532,"tokens_out":5776,"would_cite":false,"duration_ms":53028,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A token-permuting obfuscation preserves GPT-2-small's task circuit while scattering head-level traces enough to block prompt reconstruction.","keywords":["Mechanistic Interpretability","Architectural Obfuscation","Large Language Models","Privacy-Preserving Inference","Attention Mechanisms","Activation Patching","Logit Attribution","Transformer Networks"],"falsifier":"Train the same 124M-parameter architecture twice from the same initialization on the same data, once with the seeded token permutation and once without; if the diffuse patching signal and head-role reversal vanish when only the permutation is varied, the paper's central claim about obfuscation is refuted.","tokens_in":14527,"feed_emoji":"🔐","tokens_out":10148,"duration_ms":90593,"temperature":0.7,"pith_summary":"This paper asks whether a lightweight privacy defence called architectural obfuscation — scrambling a model's internal representations, here by a secret token permutation — actually stops mechanistic interpretability tools from recovering how the model works, or merely relocates the same circuits into an unfamiliar coordinate system. To answer, the authors train GPT-2-small from scratch under a fixed token permutation and apply logit-lens attribution, residual-stream patching, component patching, and head patching to an indirect-object-identification task. The central finding is a disconnect: the coarse, layer-wise computational graph that routes the correct answer survives obfuscation, while fine-grained activation patterns inside attention heads become diffuse, causal traces lose alignment with baseline semantics, and token-level logit attributions become too noisy to reconstruct a user's prompt. The paper concludes that obfuscation degrades fine-grained reverse engineering without compromising top-level task performance, giving privacy defenders and interpretability tooling a quantitative map of where the break happens.","feed_headline":"Token scrambling hides prompts but spares the model's core circuit","feed_subtitle":"Permuted tokens keep accuracy high but make logit attribution and causal patching too noisy to read.","key_machinery":"The central object is the indirect-object-identification (IOI) circuit of GPT-2-small, measured along the logit-difference axis $W_U[:,i]-W_U[:,j]$ between the correct object token and the subject token. Three causal interventions probe that axis: residual-stream patching replaces corrupted residual vectors at chosen layer/token coordinates with clean ones; block-component patching intervenes separately at attention input, attention output, and MLP output; and head patching noises or denoises individual attention heads. The obfuscation map is a fixed seeded permutation of GPT-2's vocabulary, re-indexing every token before embedding, under which the obfuscated model is trained from scratch. What the paper claims survives is the layer-wise computational graph — the sequence of residual, attention, and MLP operations moving information from the duplicated-subject token to the END token — while what changes is the activation content inside that graph.","core_discovery":"On the paper's own terms, the discovery is that a secret token-permutation obfuscation creates a disconnect between what the model computes and what causal tools can see. In the base model, the indirect-object circuit is concentrated in a few heads — heads 9.6 and 9.9 move the object information to the END token, while heads 10.7 and 11.10 subtract from the logit difference — and the important computation is finished by layer 8. In the obfuscated model, the same high-level circuit is recognizably present: the duplicated-subject position still stores the needed information, and patching still shows transfer to the END token. But the patching signal is diffuse rather than sharp, layers 9–11 continue to modify the residual stream, and the polarity of head contributions shifts, so token-level logit attribution becomes too noisy to reconstruct a user's prompt. The paper takes this as evidence that obfuscation imposes a 'representational tax' that the model pays by spreading computation over more heads and layers, preserving accuracy while impeding fine-grained reverse engineering.","pith_inferences":["An implication the authors leave implicit is that any input-space randomization strong enough to force distributed compensation should blur patching signatures; this is testable by running the same analysis on glide-reflection embedding transforms or two-dimensional shuffles instead of token permutations.","Because the obfuscated model is trained from scratch on Fineweb-Edu while the baseline is the released GPT-2-small, the cleanest attribution test would hold training data, initialization, and compute fixed and vary only the permutation; the paper's current design cannot rule out training differences as the source of the observed diffusion.","A testable extension: if the same diffuse patching signature appears under non-privacy input noise, the interpretability cost would be generic to distributed compensation rather than specific to obfuscation, linking this work to robustness studies."],"forward_implications":["Permutation-style obfuscation can preserve a model's coarse decision circuit while destroying fine-grained alignment of causal traces, so privacy defences and interpretability audits should be evaluated at both levels separately.","A server applying this obfuscation can expect global task accuracy to survive while per-token prompt reconstruction from logits degrades, the combination the paper reports.","For interpretability tooling, the results imply logit-lens attribution and head ablation still locate where information flows, but head-level causal roles in the obfuscated model cannot be assumed to match baseline roles.","The obfuscated model pays a 'representational tax': more layers stay active and computation spreads across more heads, which is the mechanism the paper offers for why accuracy survives while prompt reconstruction fails."],"supporting_citations":[{"why":"Supplies the permutation-based obfuscation design that the paper's seeded tokenizer distills.","marker":"[39]"},{"why":"Introduces the indirect-object-identification task, the logit-difference metric, and the activation-patching methodology used to locate circuits.","marker":"[34]"},{"why":"Defines the residual-stream and logit-lens decomposition that the attribution analysis builds on.","marker":"[10]"},{"why":"Provides the GPT-2-small architecture and the released HellaSwag baseline (0.2955) used for comparison.","marker":"[25]"},{"why":"Provides the Fineweb-Edu corpus on which the obfuscated model is trained.","marker":"[22]"},{"why":"Provides the HellaSwag benchmark used to set the training stopping criterion.","marker":"[38]"}],"fun_headline_variants":["Scrambled tokens keep model smart but hide its reasoning","Obfuscation preserves accuracy, destroys fine-grained insight","Secret permutations: model works, but interpretability goes blind","Token remapping shields prompts, spares computation, blocks analysis","Architectural obfuscation: accuracy intact, circuits unreadable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the baseline GPT-2-small and the obfuscated model differ only by the obfuscation map; the paper's methodology, training the obfuscated model from scratch on Fineweb-Edu and comparing against the released GPT-2-small, leaves training data and initialization uncontrolled, so any interpretability gap could come from those differences rather than from obfuscation itself.","fun_headline_variants_meta":{"raw":{"variants":["Scrambled tokens keep model smart but hide its reasoning","Obfuscation preserves accuracy, destroys fine-grained insight","Secret permutations: model works, but interpretability goes blind","Token remapping shields prompts, spares computation, blocks analysis","Architectural obfuscation: accuracy intact, circuits unreadable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000589,"raw_usage":{"total_tokens":2829,"prompt_tokens":1077,"completion_tokens":1752,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":693,"completion_tokens_details":{"reasoning_tokens":1666}},"tokens_in":693,"tokens_out":1752,"duration_ms":14696,"temperature":1.0,"reasoning_tokens":1666,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:55:51.193910+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same 124M-parameter architecture twice from the same initialization on the same data, once with the seeded token permutation and once without; if the diffuse patching signal and head-role reversal vanish when only the permutation is varied, the paper's central claim about obfuscation is refuted.","supporting_citations":[{"cited_title":"Interpretability in the wild: A circuit for indirect object identification in GPT-2","cited_arxiv_id":null,"evidence_quote":"Supplies the permutation-based obfuscation design that the paper's seeded tokenizer distills."},{"cited_title":"Scaling Monoseman- ticity: Extracting Interpretable Features from Claude 3 Sonnet","cited_arxiv_id":null,"evidence_quote":"Introduces the indirect-object-identification task, the logit-difference metric, and the activation-patching methodology used to locate circuits."},{"cited_title":"A Mathematical Framework for Transformer Circuits","cited_arxiv_id":null,"evidence_quote":"Defines the residual-stream and logit-lens decomposition that the attribution analysis builds on."},{"cited_title":"The fineweb datasets: De- canting the web for the finest text data at scale","cited_arxiv_id":null,"evidence_quote":"Provides the Fineweb-Edu corpus on which the obfuscated model is trained."}],"review_version":2}