{"arxiv_id":"2611.04782","title":"Reducing Logprob Without Causal Verification: Diagnosing SAE-Based PII Suppression in LLMs","abstract":"SAE-based feature-level interventions to suppress PII leakage in LLMs are primarily evaluated by log-probability reduction. We show that this criterion cannot distinguish feature specific suppression from a magnitude artifact: a direction-agnostic effect of sufficiently large activation perturbations. Across 36 (model, layer, PII type) cases, we compare selected SAE feature interventions with coherencematched random direction controls using logprob and rank-aware metrics. While 17 cases (47%) appear successful under logprob-only evaluation, only 1 case (2.8%) passes our multimetric framework. The rest reveal two failure modes: random directions outperform selected features, and feature selection collapse across PII types. These results indicate that existing evaluations cannot verify whether interventions target causal pathways of PII leakage. A 9,000 run causal mediation feasibility study corroborates this: the verifier separates candidates reliably, but correlation-selected candidates rarely pass it. We therefore propose a new research mission for NLP: causally grounded PII suppression through circuit-level localization and causal mediation analysis.","primary_cat":"cs.AI","cross_cats":[],"submitted_at":"2026-11-03T12:22:01+00:00","latest_version":1,"source":"arxiv","source_label":"arXiv","source_url":null,"has_pdf":true,"has_source":false,"verdict":"CONDITIONAL","confidence":"HIGH","one_line_summary":"Log-probability reduction cannot verify feature-specific PII suppression in LLMs: 17 of 36 tested interventions pass the standard test, but only 1 beats matched random-direction controls under a multi-metric criterion.","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","upvotes":0,"downvotes":0,"pith_score":0,"inbound_count":0,"_links":{"html":"https://pith.science/paper/2611.04782","verdict":"/api/v1/verdicts/2611.04782"}}