{"id":"5277449e-e43e-434b-9998-8c4e0611e714","arxiv_id":"2611.04782","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Log-probability reduction cannot verify feature-specific PII suppression in LLMs: 17 of 36 tested interventions pass the standard test, but only 1 beats matched random-direction controls under a multi-metric criterion.","lead":"This paper asks whether standard log-probability checks actually prove that AI privacy interventions remove the right internal feature. Across 36 configurations, most interventions looked successful under the old test, but almost all failed when compared with random-direction controls of equal strength.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Mediation 'verifier' is an uncalibrated single-feature, single-position noising probe: NIE_f ≈ TE for almost any feature, so the 97.3% indistinguishability rate and 'verifier separates candidates reliably' are unsupported; the Section 4 diagnostic itself is not threatened.","rationale":"The reader's CONDITIONAL verdict is justified. The Section 4 diagnostic—logprob-only evaluation cannot separate feature-specific suppression from magnitude-matched random perturbation—is well supported by held-out splits, isotropic and dictionary-aligned random controls, the Llama L16/name positive control, and threshold/alpha sensitivity; the headline arithmetic reproduces 17/36 and 1/36. The single load-bearing weakness is the secondary causal-mediation corroboration. Because NIE_f is a one-feature, one-position noising ablation and x0/x1 share the prefix, non-mediating features automatically give NIE_f ≈ TE, and the random control gives the same; the (0.5,2.0) threshold was never calibrated against a known mediator. Thus the 97.3% rate cannot be read as support for 'the verifier separates candidates reliably', and Table 1 shows only one cell with any separation. This does not weaken the central diagnostic, but it means the causal-mediation claim should be reported as a feasibility result with a sensitivity caveat, not as independent corroboration. The proposed calibration test would settle the question; until then CONDITIONAL remains the right verdict, so the reader's verdict should be unchanged.","tokens_in":25042,"tokens_out":13810,"duration_ms":91490,"concrete_test":"Calibrate the A.14 verifier on a ground-truth mediator with a positive control: take an arbitrary non-dead SAE feature g in a cell with |TE| ≥ 0.05, add β·W_dec[g] to the hidden state at the probed pre-PII token during x1 runs, with β chosen so the induced change in lp(y|x1) is comparable to the natural TE, so g is a mediator at exactly the probed position. Then run the zero-ablation protocol for g against its matched random direction. If NIE_g/NIE_rand is not below 0.5, the probe is too insensitive for the 146/150 statistic to carry the paper's causal conclusion. As a zero-cost additional check, run the same protocol on the existing positive-control feature f=32619 (Llama L16/name, Sel_lp ≈ 1995).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is the causal-mediation corroboration in §5.1/Table 1/A.14, not the Section 4 diagnostic. The redefined NIE_f = lp(y|x1, f=0) − lp(y|x0) is a single-feature, single-position noising ablation. Since x0 and x1 share all preceding text, a feature with no real downstream effect leaves lp(y|x1, f=0) ≈ lp(y|x1), so NIE_f ≈ TE; the matched random direction collapses the same way, forcing NIE_f/NIE_rand ≈ 1. The 146/150 (97.3%) rate inside the uncalibrated (0.5,2.0) window is therefore the expected null of an insensitive probe, not evidence about candidate quality. Table 1's only separable cell, Gemma name, has 11/15 inside the window (4 distinguishable) and the smallest positive TE (+0.609), while all email cells with TE > 2.4 are 100% indistinguishable; this contradicts the abstract's 'verifier separates candidates reliably'. The authors acknowledge partial circularity [b29] and attribute the bottleneck to candidate generation [b416], but no calibration on a known mediator is reported, so the data cannot distinguish 'candidates are non-causal' from 'the probe cannot detect single-feature causation'.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper challenges the dominant evaluation practice in SAE-based PII suppression, in which an intervention is deemed successful if it reduces the log-probability of the gold PII span. The authors construct 36 (model, layer, PII type) cells across three 8B/9B LLMs fine-tuned on a synthetic medical corpus, select a putative PII feature per cell by logprob reduction on a discovery split, and compare hidden-state interventions along that feature's decoder direction against coherence-matched random controls (isotropic and dictionary-aligned) using logprob, a rank-aware extraction-cost proxy, and selectivity ratios. They report that 17/36 cells (47%) pass a logprob-only gate, while only 1/36 (2.8%; Llama L16/name) passes the conjunctive multi-metric criterion. The two failure modes are random-dominant non-selective effects and feature-selection collapse across PII types. The criterion is supported by a positive control (Sellp ≈ 1995, stable across nine templates), threshold sensitivity analyses, an alpha sweep, and a downstream-utility check. A second study runs 9,000 noising-style causal mediation probes on activation-rate-selected candidates; the authors claim it corroborates the diagnostic and places the bottleneck in candidate generation rather than verification.","tokens_in":25287,"tokens_out":14779,"duration_ms":89060,"significance":"If the Section 4 diagnostic is correct, it is a timely negative result for interpretability-based privacy: the shared evaluation signal of recent SAE-based PII suppression work (PrivacyScalpel, ProPILE, PII-Scope) cannot by itself establish feature-specific suppression. The experimental design is considerably more careful than is typical in this literature: mutually disjoint discovery/validation/synthetic-SAE corpora, two random-control families including same-SAE dictionary directions, a genuinely selective positive control, and sensitivity analyses over thresholds and intervention strength. The promised release of code, corpora, and trained SAEs is a further strength. The causal-mediation feasibility study is the weak link: the proposed verifier is an uncalibrated single-feature, single-position noising probe, and the paper's claim that it 'separates candidates reliably' is contradicted in direction by its own Table 1. The Section 4 claims do not depend on the mediation study, but the abstract's corroboration claim and the conclusion that the bottleneck is candidate generation do.","major_comments":[{"comment":"The NIE estimator is an uncalibrated single-feature, single-position noising probe, and the 97.3% 'indistinguishable' rate is largely mechanical. NIE_f = lp(y|x1, f=0) − lp(y|x0); since x0 and x1 share all preceding text and the intervention zero-ablates one feature at the single token immediately before the PII span, any feature whose one-token ablation has a negligible downstream effect yields lp(y|x1, f=0) ≈ lp(y|x1) and hence NIE_f ≈ TE. The matched random direction collapses identically (NIE_rand ≈ TE), forcing NIE_f/NIE_rand ≈ 1 into the (0.5, 2.0) window for nearly every candidate. The authors acknowledge the selection-circularity limitation (Limitations, 'Circular selection design'; §3) but do not address this probe-sensitivity confound: no calibration on a feature known — by construction or by an independent method — to mediate PII generation is reported anywhere in Appendix A.14. Without such calibration, '146/150 (97.3%) are indistinguishable' cannot support the abstract's claim that correlation-selected candidates 'rarely pass' causal verification, because a completely insensitive probe produces exactly this pattern. The data as analyzed cannot distinguish 'candidates are non-causal' from 'the probe cannot detect single-feature mediation at one position.'","section":"§5.1, Table 1, Appendix A.14","authors_acknowledge":"We note that this design is itself partially circular, since logprob serves both as the selection criterion and as the evaluated met- ric; Section 5.1 therefore revisits the same question with candidates selected independently of logprob.","authors_acknowledge_p":0.76},{"comment":"The direction of the observed separability pattern argues against the claim that 'the verifier separates candidates reliably.' The only cell in which any candidate is separable is Gemma name (11/15 inside the window, i.e., 4/15 distinguishable), which has the smallest positive TE (+0.609). The three email cells, which carry the strongest memorization signal (TE = +2.580, +3.560, +2.466 across the three models), are 100% indistinguishable. A verifier that separates candidates by 'whether genuine memorization is present' should show more separability where memorization is stronger, not less. The observed ordering is precisely what one expects if the individual ratios are noisy TE-over-TE quantities: when TE is large the ratio is stably near 1, and when TE is small sampling noise pushes a few ratios outside the window. The Section 5 conclusion that 'the bottleneck is candidate generation' therefore does not follow from the reported data: the alternative that the probe lacks sensitivity to single-feature mediation fits the same data at least as well.","section":"§5.1 / Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The mediation analysis reports no uncertainty quantification, and the (0.5, 2.0) window is never calibrated. With 50 pairs per cell, the means NIE_f and NIE_rand are subject to nontrivial sampling noise, yet the Indist. percentages in Table 1 are treated as exact. The window is also intrinsically conservative: a feature that genuinely mediates 50% of TE yields NIE_f/NIE_rand = 0.5, exactly on the boundary, so any mediation below roughly 50% is 'indistinguishable' by construction, regardless of probe quality. The mediation fraction MF_f is defined in §5.1 but never reported; reporting MF_f with confidence intervals, or a sensitivity analysis on synthetic mediators with known effects, is the minimum needed to make the feasibility study interpretable as evidence about candidate quality.","section":"Appendix A.14","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The sentence 'The sole exception, Gemma-2-9B/name (11/15), is precisely the one combination exhibiting a positive memorization signal' is contradicted by the paper's own Table 18 (Gemma email, TE = +2.580); the intended wording is the Table 1 note's 'the only non-email cell with a positive memorization signal,' and the main text should match it.","section":"§5.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Two cells (Llama L24/phone, Llama L24/email) are labeled 'inactive' with undefined selectivity; a sentence stating explicitly that such cells fail every gate and remain in the 36-cell denominators of the 17/36 and 1/36 counts would remove ambiguity for readers.","section":"Table 9","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Cell-level means are reported without dispersion (e.g., over the 450 evaluation contexts or the three random-control seeds). Given that several conclusions hinge on ratio-based selectivity values, including Qwen L27/id Sellp = 115.1 and Llama L16/name Sellp ≈ 1995 with the random denominator at the measurement floor, reporting per-cell standard errors or the distribution of per-pair values would aid assessment.","section":"Table 9 / Appendix A.12","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The alpha-robustness claim is established on only three representative cells and two templates (T1, T5), and one of the three cells (Qwen L18/name) flips from selective to random-dominant as alpha varies; the main text should state this scope where robustness to intervention strength is claimed.","section":"Appendix A.12 / Section 4","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Bottom line: the Section 4 diagnostic is the real contribution and it holds. The mediation run in §5.1 does not, and the abstract overstates it. I would send this to review, but only with §5.1 either recalibrated or explicitly downgraded to an uncalibrated pilot.\n\nWhat is new: the 36-cell grid that applies random-direction controls to SAE-based PII suppression. The random-baseline critique itself is not new—Korznikov, Ronge, Heap, and Méloux have made the general point—but the instantiation is. Logprob-only evaluation passes 17/36 cells; a conjunctive criterion with matched controls passes 1/36. The two failure modes, random-dominant and selection collapse, are documented cleanly, with a positive control (Llama L16/name, Sel_lp ≈ 1995, no utility loss) showing the criterion is not impossibly strict. The controls are matched for magnitude, span-level coherence, and dictionary alignment, with sensitivity analyses on thresholds and α. Code, data, and SAEs are released, and I checked the headline arithmetic in Table 9: 17 and 1 are right.\n\nWhere it wobbles: §5.1. The noising-style NIE_f is a single-feature, single-position zero-ablation. Since x0 and x1 share the preceding text, standard patching is identically zero, and the substitute probe is never calibrated. For almost any feature, ablating it at one position leaves lp(y|x1,f=0) ≈ lp(y|x1), so NIE_f ≈ TE; the random control collapses the same way. The 97.3% indistinguishability rate is close to the expected null of an insensitive probe, not evidence about candidate quality. Table 1 makes the problem visible: the only separable cell has 11/15 inside the window and the smallest positive TE, while the email cells with TE > 2.4 are 100% indistinguishable. The authors acknowledge circularity and scope their conclusion elsewhere ([b29], [b123]), and they report the numbers honestly, but the abstract's 'verifier separates candidates reliably' is not supported. The mediation section should not be cited as causal verification.\n\nWho gets value: people working on SAE interpretability, privacy defenses, and evaluation methodology. The Section 4 result is reproducible and relevant, and it deserves a serious referee. Recommendation: accept with revision—recalibrate the mediation probe on a known mediator or soften the causal-verification claims.","headline":"The 36-cell random-control diagnostic is sound and worth publishing; the mediation 'verifier' in §5.1 is an uncalibrated probe and should not be read as causal corroboration.","tokens_in":25996,"tokens_out":3415,"duration_ms":30623},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"SAE-based PII suppression judged only by log-probability reduction cannot verify any PII-specific effect. Across 17 apparent successes, only one cell survives a multi-metric random-control test.","keywords":["sparse autoencoders","PII suppression","log-probability evaluation","random baseline controls","causal mediation analysis","feature ablation","LLM memorization","mechanistic interpretability"],"falsifier":"Repeat the mediation analysis of Appendix A.14 on the three email cells, where the memorization signal is strongest ($\\mathrm{TE} > 2.4$ on all three models), applying the zero-ablation at every token of the PII span instead of only at the token immediately before it, using the same counterfactual pairs and activation-rate candidate selection. If any correlation-selected feature then yields $\\mathrm{NIE}_f/\\mathrm{NIE}_{\\text{rand}}$ clearly outside the (0.5,2.0) indistinguishability window, the claim that candidate generation is the bottleneck and that logprob-selected features are causally inert would be refuted for that cell.","tokens_in":28961,"feed_emoji":"🛡️","tokens_out":6607,"duration_ms":134907,"temperature":0.7,"pith_summary":"This paper tries to establish that the standard measure of success for SAE-based PII suppression in LLMs, the reduction of the gold PII token's log-probability, cannot by itself certify that an intervention targeted a PII-specific mechanism: any sufficiently large hidden-state perturbation will lower the probabilities of subsequent tokens regardless of direction. Across 36 (model, layer, PII type) cells, 17 (47%) look successful under a logprob-only gate ($\\Delta_{\\text{lp}} \\leq -0.3$), but only 1 (2.8%, Llama L16/name) survives a conjunctive multi-metric gate that also requires a rank-aware extraction-cost improvement and 5-fold selectivity over coherence-matched random and dictionary-aligned controls. The failures separate into two patterns: random directions frequently outperform the selected feature, and the same top-1 feature can be selected for all four PII types while producing type-asymmetric effects. A 9,000-run causal mediation feasibility study corroborates the diagnosis, finding 146 of 150 activation-rate-selected candidates statistically indistinguishable from a random control. If the paper is right, previously reported SAE-based PII-suppression results are unverified as feature-specific, and logprob-based selection and evaluation should be replaced by causal mediation analysis and circuit-level localization.","feed_headline":"One in 36 PII suppression interventions beats random controls","feed_subtitle":"Log-probability drops alone don't prove feature-specific suppression; the paper argues PII defenses need causal verification.","key_machinery":"The load-bearing device is the coherence-matched random direction control. The selected feature intervention $h'_t = h_t + \\alpha z_t[f] W_{\\text{dec}}[f]$ is compared with $h'_t = h_t + \\alpha m_t r$, where $r$ is an isotropic random unit vector (or a normalized decoder direction of another feature from the same SAE), $m_t = \\|z_t[f]W_{\\text{dec}}[f]\\|_2$, and the same direction is shared across all intervened positions so that span-level direction coherence is preserved. Because magnitude, position pattern, and coherence are matched, the only remaining difference is the direction, which is exactly the carrier of feature-specificity. Selectivity is the ratio of effect magnitudes, $\\text{Sel}_{\\text{lp}} = |\\Delta^{\\text{target}}_{\\text{lp}}| / |\\Delta^{\\text{random}}_{\\text{lp}}|$ and analogously $\\text{Sel}_{\\text{br}}$, and the clean-strong label additionally requires $\\Delta_{\\text{lp}} \\leq -0.3$ and $\\Delta_{\\text{br}} \\geq +1.0$. In the mediation leg, the central identity is $\\mathrm{NIE}_f = \\mathrm{lp}(y|x_1, f=0) - \\mathrm{lp}(y|x_0)$ against $\\mathrm{TE} = \\mathrm{lp}(y|x_1) - \\mathrm{lp}(y|x_0)$, computed with a single-feature zero-ablation because the standard activation-patching NIE is shown to be identically zero when $x_0$ and $x_1$ share all preceding context.","core_discovery":"On the paper's own terms, the central claim is that using the log-probability of the gold PII span both to select SAE features and to judge whether their ablation or steering suppressed leakage is circular and non-specific: any sufficiently large directional perturbation of the hidden state reproduces the headline effect. The paper operationalizes specificity as beating a coherence-matched control by at least a factor of five on both the logprob metric and a rank-aware extraction-cost proxy, where the control preserves intervention strength, token-wise magnitudes, and span-level direction coherence and differs only in direction. Across three 8B/9B LoRA-fine-tuned models, only one of 36 configurations, Llama L16/name, passes all four conjunctive criteria ($\\Delta_{\\text{lp}} \\leq -0.3$, $\\Delta_{\\text{br}} \\geq +1.0$, $\\text{Sel}_{\\text{lp}} \\geq 5$, $\\text{Sel}_{\\text{br}} \\geq 5$), and this collapse from 47% to 2.8% survives threshold and intervention-$\\alpha$ ($\\alpha \\in \\{-1,-2,-3,-4,-5\\}$) robustness checks. The failures are non-exclusive: random-dominant cells (e.g., Gemma L20/name, where 20/20 dictionary-aligned controls suppress more than the selected direction) and feature-selection-collapse cells (e.g., Llama L8, where one feature is selected for all PII types yet its effect is 2.3x stronger on name than on the others at matched activation magnitudes). A feasibility study with 9,000 causal patching runs, using a noising-style single-feature zero-ablation $\\mathrm{NIE}_f = \\mathrm{lp}(y|x_1, f=0) - \\mathrm{lp}(y|x_0)$ against $\\mathrm{TE} = \\mathrm{lp}(y|x_1) - \\mathrm{lp}(y|x_0)$, finds 146 of 150 (97.3%) activation-rate-selected candidates within a factor of two of the random control, with the sole exception being Gemma-2-9B/name (11 of 15 distinguishable); the paper concludes that the mediation verifier works and that the bottleneck is candidate generation.","pith_inferences":["Editorial extension: the coherence-matched random control could become a standard sanity check for other SAE steering applications (refusal, honesty, safety) that evaluate success by likelihood changes, because the non-specificity critique is not unique to PII.","Editorial extension: the paper's own Table 1 suggests the mediation probe may be underpowered: the only distinguishable cell has a modest positive memorization signal, while the strongly memorized email cells (TE > 2.4) yield no distinguishable candidate at the single-token probe; a multi-token ablation is the natural next test.","Editorial extension: strong email memorization coexisting with no single distinguishable feature is consistent with a distributed causal pathway, which would explain why single-feature interventions fail and would motivate circuit-level rather than single-feature editing."],"forward_implications":["SAE-based PII suppression studies that report only logprob or leakage-rate reductions cannot be taken as evidence of feature-specific intervention; the 35 non-clean-strong cells are consistent with generic magnitude artifacts.","The conclusion is threshold-independent: loosening the gates still yields at most 3/36 (8.3%) clean-strong cells, so the failure of logprob-only evaluation is not an artifact of strict cutoffs.","Genuine feature-specific effects exist and are detectable: Llama L16/name passes all criteria across all nine templates, including out-of-distribution Korean and JSON formats, and preserves downstream accuracy on PubMedQA, MedQA, and MMLU.","Causal mediation analysis is implementable for PII leakage with 9,000 runs and correctly separates cells by whether memorization is present; the bottleneck it reveals is causally motivated candidate generation, to be addressed by circuit-level localization.","Extraction-attack benchmarks should be used alongside feature-specificity diagnostics, not instead of them, since extraction-rate drops can stem from general generation degradation."],"supporting_citations":[{"why":"PrivacyScalpel, the representative SAE-based PII suppression method whose logprob/leakage-rate evaluation is the target of the paper's critique.","marker":"Frikha et al., 2025"},{"why":"Cited as part of the standard practice of suppressing or evaluating PII leakage via PII-token log-probability.","marker":"Kim et al., 2023"},{"why":"PII-Scope benchmark, supplies the leakage-rate evaluation signal the paper argues cannot verify feature-specificity.","marker":"Nakka et al., 2024"},{"why":"Shows SAE features may not beat random baselines; motivates the coherence-matched random control used in the selectivity experiments.","marker":"Korznikov et al., 2026"},{"why":"Provides the causal mediation analysis (NIE) framework adapted in the feasibility study, replacing patching with a noising-style ablation.","marker":"Vig et al., 2020"},{"why":"Supplies the causal-tracing noising intervention that the single-feature zero-ablation conceptually corresponds to.","marker":"Meng et al., 2022"},{"why":"Path patching, proposed as the method for circuit-level localization of the causal pathways behind PII leakage.","marker":"Wang et al., 2022"},{"why":"Provides the association-versus-intervention framing (ladder of causation) that structures the argument against correlation-based evaluation.","marker":"Pearl, 2009"}],"fun_headline_variants":["Only one PII suppression intervention in 36 survives random-direction controls","Logprob-only PII checks cannot distinguish feature suppression from magnitude artifacts","Random controls cut apparent SAE PII suppression success from 47% to 2.8%","SAE PII suppression needs causal verification, not logprob-only checks","A 9,000-run mediation study says feature selection, not verification, is the bottleneck"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The weakest load-bearing assumption is that zero-ablating one SAE feature at the single token immediately before the PII span is a sensitive enough causal probe to detect a genuine PII mediator; the authors adopt this noising-style probe because the standard activation-patching NIE is identically zero at that position, but they do not calibrate its sensitivity, so the 97.3% 'indistinguishable from random' rate could be a mechanical consequence of a weak probe rather than evidence that the features are causally inert.","fun_headline_variants_meta":{"raw":{"variants":["Only one PII suppression intervention in 36 survives random-direction controls","Logprob-only PII checks cannot distinguish feature suppression from magnitude artifacts","Random controls cut apparent SAE PII suppression success from 47% to 2.8%","SAE PII suppression needs causal verification, not logprob-only checks","A 9,000-run mediation study says feature selection, not verification, is the bottleneck"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.001436,"raw_usage":{"cost":0.0014355432,"is_byok":false,"cost_details":{"upstream_inference_cost":0.0014355432,"upstream_inference_prompt_cost":3.77832e-05,"upstream_inference_completions_cost":0.00139776},"total_tokens":8976,"prompt_tokens":1296,"completion_tokens":7680,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":1101,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":7588}},"tokens_in":1296,"tokens_out":7680,"cost_source":"openrouter_usage","duration_ms":51996,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":7588,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-04T13:23:20.769976+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Repeat the mediation analysis of Appendix A.14 on the three email cells, where the memorization signal is strongest ($\\mathrm{TE} > 2.4$ on all three models), applying the zero-ablation at every token of the PII span instead of only at the token immediately before it, using the same counterfactual pairs and activation-rate candidate selection. If any correlation-selected feature then yields $\\mathrm{NIE}_f/\\mathrm{NIE}_{\\text{rand}}$ clearly outside the (0.5,2.0) indistinguishability window, the claim that candidate generation is the bottleneck and that logprob-selected features are causally inert would be refuted for that cell.","supporting_citations":[{"cited_title":"Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP , pages =","cited_arxiv_id":null,"evidence_quote":"PrivacyScalpel, the representative SAE-based PII suppression method whose logprob/leakage-rate evaluation is the target of the paper's critique."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the causal-tracing noising intervention that the single-feature zero-ablation conceptually corresponds to."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}