{"id":"396cf1af-f474-4d73-9aa9-52400f1a3973","arxiv_id":"2607.20379","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Reconstruction scores do not certify individual claims in activation explanations; co-adapted private codes can carry the score, and target-side training (RECAP) makes designated content verifiably decodable.","lead":"This paper shows that reconstruction-based explanations of neural activations can earn high scores while most of their individual claims are ungrounded or false, and that co-trained \"private codes\" can carry the score. It introduces claim-level audit protocols and a target-side training method, RECAP, that makes designated content genuinely decodable and checkable by independent probes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RECAP's scale transfer rests on a target-design rule that is only stated after a failed target; no prospective criterion is given, so success may reflect post-hoc selection rather than a general method.","rationale":"The reader's weakest assumption is exactly the target-design rule. I agree. The paper's central contributions are the audit (structurally sound) and RECAP (promising repair). The audit side is well-supported: the released-system gist finding, the synthetic private-code discovery with evaluator swap, and the collapse/drift analyses are internally consistent and controlled. The RECAP side is strong in the sandbox (5/5 runs, independent probes) but at scale every headline number is tied to the choice of targets. The K=512 naive failure shows target choice matters, and the rule 'loss cannot be minimized without genuine discrimination' is a useful heuristic but not a prospectively operationalizable criterion as written. The paper's own limitation statement admits a poorly chosen target can report success while decoding nothing. Thus the scale transfer claim is conditional on a selection procedure that is not fully specified. A pre-registered replication on a new model would settle whether the rule is sufficient. This does not change the verdict: the reader already assigned CONDITIONAL. No other concern seems more load-bearing: the low ~2% grounding is explicitly a sensitivity-limited lower bound; the synthetic codes are robust; the verbalizer truth at scale is honestly reported as partial; the adversarial probe test is well-controlled. Therefore verdict stays CONDITIONAL.","tokens_in":23084,"tokens_out":5303,"duration_ms":50056,"concrete_test":"Pre-register a target-selection checklist on a new, held-out setting (e.g., Pythia-410M or continued pretraining on a different corpus): (a) for each candidate target, compute the loss of the best constant predictor on a held-out split and require it to be above a pre-set threshold (e.g., within 20% of the random-guess loss); (b) require class balance or use the class-balanced loss of §5.3; (c) require a pilot probe AUC > 0.9 on a small run. Then run RECAP with these targets and no further iteration on target choice. If a fresh probe reaches AUC > 0.95 and the held-out LM tax stays within the reported seed spread (95% CI [−0.015,+0.034] for K=64), the rule is prospective. If a target passes the checklist but fails independent probes, the rule is insufficient and the scale claim is conditional on retrospective selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim: RECAP makes designated internal content probe-decodable on Pythia-160M (§5.3, Table 1), enabling probe-based verification (AUC 0.965, §6). This transfer depends on the 'design rule' of §5.3: 'use targets whose loss cannot be minimized without genuine discrimination.' The rule is introduced after the K=512 naive presence head failed by predicting 'absent' for rare targets. No a priori operationalization is provided: how does one identify such targets before training? Class-balanced loss fixes the one documented failure, but the rule remains underspecified; the paper itself concedes that RECAP 'depends on choosing a good target: a poorly chosen one can report success while decoding nothing' (Discussion, Limitations). Because all scale results — probe AUC, verbalizer truth 0.44–0.46, adversarial robustness — were obtained under this retrospectively stated rule, a new application must either guess targets or iterate until fresh probes succeed. If the rule cannot be made prospective, the method's transfer claim is conditional, not a settled recipe. This is not an internal inconsistency; it is a missing specification that bears directly on the central safety/interpretability promise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the reconstruction test used by natural-language autoencoders (NLAs) is structurally insensitive to individual false claims in generated explanations, and that passing the test is therefore not evidence of claim-level faithfulness. It supports this with two empirical demonstrations: on a released Qwen-2.5-7B verbalizer/reconstructor pair, explanations reconstruct well above chance while only a small fraction of audited specific claims are reconstruction-dependent (the score tracks gist rather than specifics); and in an exact-ground-truth synthetic sandbox, co-trained verbalizer/reconstructor pairs develop private codes—grounded but false wording—in 5/5 standard-recipe runs. The paper then proposes RECAP, a method that co-trains linear auxiliary heads on external targets during target-model training, so that designated content remains linearly decodable from chosen hidden states. In the sandbox, RECAP gives perfect fresh-probe decodability and propagates to truthful fresh verbalizers at negligible cost. On continued pretraining of Pythia-160M, designated content becomes probe-decodable (AUC 0.95–1.00) under a stated 'target-design rule,' with a +0.010-nat tax for 64 balanced targets; a fresh probe scores the verbalizer's true claims above false ones at AUC 0.965 versus 0.823 for the control, and this survives a report-space adversary. The paper is notably candid about limitations: decodability is not verbalizability, the method depends on target choice, a","tokens_in":23390,"tokens_out":5574,"duration_ms":53834,"significance":"If the main claims hold, the paper makes a substantial contribution to interpretability and AI-safety methodology. The audit protocols—claim-level flips with validity controls, the grounded-vs-true cross, the evaluator swap—are well designed, and the exact-ground-truth sandbox gives unusually strong causal evidence that the standard reconstruction recipe can fail in a non-obvious way. The use of fresh independent probes rather than co-trained heads for all decodability measurements is a methodological strength. The adversarial lie-ladder, held-out band null, frozen-threshold anti-circularity check, and class-balanced loss analysis are also careful. The central theoretical point—that reconstruction rewards sufficiency, not claim truth—is sound and important. The main limitation is that the method's transfer to real models rests on an underspecified target-design rule and on a single small model (Pythia-160M); the paper is open about this, but the methodological recipe as written is not yet fully reproducible for new applications.","major_comments":[{"comment":"The RECAP scale transfer depends on the target-design rule stated in §5.3: 'use targets whose loss cannot be minimized without genuine discrimination.' This rule is introduced only after the K=512 naive presence head failed, and no prospective, operationalizable criterion is given for identifying such targets before training. The paper itself concedes in §8 that 'it depends on choosing a good target: a poorly chosen one can report success while decoding nothing.' This is load-bearing because all scale results—Table 1 probe AUCs, §5.4 verbalizer truth, §6 monitor AUC and adversarial robustness—were obtained under this retrospectively stated rule. As written, the rule is unfalsifiable in practice: if a chosen target fails, the method can always attribute the failure to poor target selection. The revision should provide a pre-hoc diagnostic or formal condition for when an auxiliary target's","section":"§5.3, Eq. (3), Table 1; §8 Limitations"},{"comment":"The scale verbalizer still shows a substantial grounded-minus-true gap (+0.33 after RECAP, versus +0.42 control) and per-word truth of only 0.44–0.46; the paper honestly says the content is 'stated more truthfully, not code-free.' This means the sandbox 'codes vanish' result does not fully transfer, and the practical guarantee at scale is explicitly probe decodability, not verbalizer truth. I do not count this as a fatal error because the paper frames the probe as the reliable readout and the adversarial monitor is measured on probe AUC. But the title and abstract should not leave the impression that RECAP makes free-form explanations faithful at scale; the distinction between 'decodability' and 'verbalizability' should be emphasized in the framing, perhaps in the title. If the contribution is intended as a safety mechanism, the scope of the guarantee (designated content only, probe-veri","section":"§5.4, Table 13; §6"}],"minor_comments":[{"comment":"The phrase 'as few as ∼2%' is slightly ambiguous: the base system gives 4.2% at τ=0.02, 2.1% at τ=0.05, 1.6% at τ=0.10. The paper correctly calls these sensitivity-limited lower bounds, but the abstract's wording could be read as an upper bound. Consider saying 'at most a few percent' or quoting the τ=0.05 number.","section":"Abstract and §3.1"},{"comment":"The auxiliary-weight sweep appears to be single-seed per point (Table 12 caption). For the crucial claim that identity targets remain readable at w=0.03 with tax within noise of zero, a paired multi-seed confidence interval would be more persuasive. The paper gives one for K=64 balanced at w=0.1, but not for the low-weight identity case.","section":"§5.3 and Appendix Table 12"},{"comment":"The erosion of decodability under head-free fine-tuning within 250 steps is a major deployment constraint and is only in an appendix. I suggest moving the sentence 'RECAP is a running cost, not a one-time purchase' into the main text, since safety users will need it to design monitoring schedules.","section":"Appendix C, Table 3"},{"comment":"Figure 3(b) contains a small grammar error in the generated explanation ('a acorn' instead of 'an acorn'). This does not affect the results but should be fixed for publication.","section":"Figures"},{"comment":"The offset-8 target is reported as unrecovered with a base-rate caveat (true designated token in only 2.2% of taps). This is appropriately transparent, but the reader may wonder whether it weakens the claim that 'designated content' is decodable. Please add one sentence explaining whether the identity target at offset−8 is considered a failed target under the §5.3 design rule, or simply underpowered.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is unusually honest and technically careful; the core critique of reconstruction-based faithfulness testing is well supported. My main concern is the one raised in the stress test: the target-design rule in §5.3 is post hoc and underspecified, and it is load-bearing for the scale-transfer and safety claims. I would not reject, but the revision should make the target-selection criterion prospective and preferably add at least one confirmatory run at a second model scale or architecture. I also think the title slightly overpromises relative to the decodability-not-verbalizability result at scale; this is fixable by framing. The paper's transparency about withdrawn contaminated results in Appendix G is commendable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this for the audit, not for the fix. The claim-level counterfactual audit of reconstruction-scored explanations is genuinely new and well executed: minimal-pair flips on a released Qwen-2.5-7B verbalizer leave reconstruction essentially unchanged (~2% grounded claims at standard thresholds), transform controls show the score tracks gist rather than specific wording, and the exact-ground-truth sandbox shows co-trained verbalizer/reconstructor pairs develop private codes in 5/5 runs. The evaluator swap cleanly distinguishes co-adapted convention from genuine grounding. The paper also reports honestly: trained controls that fail, partial verbalization at scale (truth 0.44–0.46 vs near-zero control), one 160M model, mostly single seeds, and an explicit limitation list that names the target-design problem.\n\nNow the soft spots, in proportion. The target-design rule of §5.3 is exactly as retrospective as the stress-test note says. It appears only after a naive K=512 head fails, and the paper itself concedes that a poorly chosen target can report success while decoding nothing. No prospective operationalization is given. That makes the transfer claim conditional; it doesn't undermine the audit, but it means the method is not yet a recipe. Second, there is no public code or commit hash despite reproducibility claims. Third, the scale evidence rests on a single model and a small effect; the verbalizer gain is real but weak, and the target-design caveat could interact with it.\n\nOn balance, the central critique holds up and is worth taking seriously. The adversarial result — reconstruction-score lie penalty suppressed ~87% while a fresh probe still flags lies (AUC 0.95 vs 0.51) — is a solid demonstration that decodability, not the score, is the usable truth signal. I would send this to a serious referee. The main revision demand should be: state the target-design rule prospectively, or explicitly frame the method as a family of designs with open search, and release the code. That is an addressable gap, not a fatal flaw.","headline":"The audit is the real contribution; the scale-transfer fix is promising but conditional on a target-design rule the paper only states retrospectively.","tokens_in":23819,"tokens_out":3450,"would_cite":true,"duration_ms":30168,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reconstruction test for AI explanations does not certify individual claims: the paper shows high reconstruction is achieved while most specific claims are ungrounded, and proposes RECAP, which trains the target model to keep designated co","keywords":["natural-language autoencoders","activation explanations","reconstruction score","faithfulness","interpretability","probe-based monitoring","RECAP","private codes"],"falsifier":"Run RECAP on a pretrained model using a target selected purely prospectively from the stated rule (before seeing whether probes read it); if a fresh probe's AUC on the designated content is no better than the 0.77–0.80 control baseline, or if a fresh verbalizer's truth rate on that content is not above the control's 0.06, the claim that RECAP makes designated content verifiable at scale is falsified. In the sandbox, a single run in which a fresh verbalizer shows a grounded-minus-true gap comparable to the standard recipe's 0.30–0.80 would falsify the claim that RECAP eliminates co-adapted priv","tokens_in":22968,"feed_emoji":"🧠","tokens_out":5668,"duration_ms":44740,"temperature":0.7,"pith_summary":"The paper argues that the reconstruction test used by natural-language autoencoders—regenerating a hidden activation from an explanation and treating the similarity as faithfulness—is structurally insensitive to individual false claims. A flipped claim is penalized only if the reconstruction drops, and in practice most specific claims are not reconstruction-dependent: on a released Qwen-2.5-7B verbalizer, explanations reconstruct at r~0.84 while only about 2% of audited specific claims are grounded at the standard threshold. Under exact synthetic ground truth, the standard co-training recipe develops co-adapted private codes (false wording the reconstruction depends on) in 5/5 runs, so the test can be passed by gist and by private conventions rather than faithful content. The paper's repair is RECAP: during training, linear heads predict external targets from hidden states, making designated content decodable. In a sandbox this yields claim-faithful fresh verbalizers in 5/5 runs at negligible cost, and on a pretrained Pythia-160M it makes the content probe-decodable, raising fresh-verbalizer truth from near zero to 0.44–0.46 and letting a probe catch lie edits at AUC 0.95.","feed_headline":"Reconstruction scores miss most false claims in AI explanations","feed_subtitle":"On a released verbalizer only ~2% of specific claims are reconstruction-dependent; RECAP makes content probe-checkable.","key_machinery":"The central object is the reconstruction objective J_read = max cos(AR(AV(h)), h), which rewards sufficiency but not per-claim truth. The paper's audit instruments are the flip audit (per-claim grounding = reconstruction drop under a minimal-pair edit) and the grounded-vs-true cross with an evaluator swap (an independent reconstructor), which exposes co-adapted private codes. RECAP replaces J_read-side training with a model-side objective J_model = min LLM(theta) + w L_phi(h(theta), y), where linear heads predict external targets; fresh probes then verify decodability. The scaling condition is a target-design rule: auxiliary targets must be chosen so their loss cannot be minimized without ge","core_discovery":"The paper's central discovery is that the reconstruction score measures gist and co-adaptation, not claim-level truth. On the released system, meaning-preserving paraphrase keeps the score while masking content words destroys it, and an append-and-rank control recovers a known-encoded topic at only 28 percent raw, showing the score tracks overall meaning. In the sandbox, the co-trained pair uses grounded-but-false wording (private codes) that an independent evaluator cannot read; the grounded-minus-true gap is 0.30–0.80 in 5/5 standard runs and collapses to 0.08–0.23 under an evaluator swap. The author's proposed fix, RECAP, changes the target model rather than the reader: linear heads co-tr","pith_inferences":["The claim-level audit standard the paper introduces is portable: any reconstruction-scored explainer, not just the released Qwen-2.5-7B pair, could be audited with the same flip protocol and evaluator swap, and the ~2% grounded-fraction figure is likely a lower bound because the instrument is sensitivity-limited.","The target-design rule is the method's open condition. If it cannot be stated prospectively—what property makes a target's loss impossible to minimize without discrimination—then RECAP's success at scale depends on post-hoc validation, and a principled selection rule would be needed before relying on it in new settings.","The paper's distinction between storage (decodability) and verbalizability suggests that even successful interpretation pipelines may need probes rather than prose as the trustworthy interface; a testable extension is whether a generation-time objective that directly rewards truth of generated tokens, rather than reconstruction, would close the 0.44–0.46 gap.","For frontier models, the open scaling question is whether external self-supervised targets (like the offset-8 token) remain sufficient to keep designated content decodable as model scale grows; the paper did not vary model scale beyond 160M."],"forward_implications":["Individual claims in reconstruction-scored explanations cannot be treated as verified by a high reconstruction number; claim-level audits or probe-based checks are required.","The released Qwen-2.5-7B verbalizer's high score should be interpreted as gist-level informativeness, not evidence that its specific statements are grounded in the activation.","For safety monitoring, RECAP provides a designated-content readout that an independent probe can check; false claims about the designated content are flagged at high AUC even when the reconstruction score is gamed.","Because decodability erodes within a few hundred steps of head-free fine-tuning, interpretability guarantees from such supervision must be maintained continuously, not set once.","The grounded-vs-true cross and evaluator swap give future deployments a cheap way to test whether an NLA's explanations carry private codes."],"fun_headline_variants":["Reconstruction scores track gist, not claim-level truth","RECAP makes AI internal content independently checkable","AI explanation scores miss ~2% of specific false claims","A new method catches lies that reconstruction misses"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method's transfer to scale rests on the rule that auxiliary targets must be chosen so their loss cannot be minimized without genuinely discriminating, and the paper identifies this rule only after a naive target failed, without giving an a priori criterion for picking such targets.","fun_headline_variants_meta":{"raw":{"variants":["Reconstruction scores track gist, not claim-level truth","RECAP makes AI internal content independently checkable","AI explanation scores miss ~2% of specific false claims","A new method catches lies that reconstruction misses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000438,"raw_usage":{"total_tokens":2154,"prompt_tokens":925,"completion_tokens":1229,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":1167}},"tokens_in":669,"tokens_out":1229,"duration_ms":8278,"temperature":1.0,"reasoning_tokens":1167,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T09:58:52.208394+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RECAP on a pretrained model using a target selected purely prospectively from the stated rule (before seeing whether probes read it); if a fresh probe's AUC on the designated content is no better than the 0.77–0.80 control baseline, or if a fresh verbalizer's truth rate on that content is not above the control's 0.06, the claim that RECAP makes designated content verifiable at scale is falsified. In the sandbox, a single run in which a fresh verbalizer shows a grounded-minus-true gap comparable to the standard recipe's 0.30–0.80 would falsify the claim that RECAP eliminates co-adapted priv","supporting_citations":[],"review_version":1}