{"id":"fcdbc92f-3cf4-4711-a16f-b9b5b4f0c43f","arxiv_id":"2602.01425","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Deception probes trained with taxonomy-specific prompts appear to beat a universal probe only because the best prompt is selected per dataset after evaluation; a priori matching is claimed but not demonstrated.","lead":"An AI-safety paper tests whether one internal 'deception probe' can catch all kinds of lying and finds that different deception types respond to different training prompts — but the headline gain (+0.108 AUC) comes from choosing the best probe after seeing each test set, so the practical promise of targeted detectors is not yet established.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Oracle-selected best-of-16 taxonomy probes cannot support the a priori achievability claim; a held-out or permutation control is needed.","rationale":"The reader's CONDITIONAL verdict aligns with my stress-test. The central claim—that targeted probes improve detection when the deception type is known a priori—rests entirely on post-hoc oracle selection (best of 16 taxonomy prompts per dataset). The paper transparently states this selection procedure, but it is not a valid basis for the a priori prescription. The absence of a control using the 34 framing variations (which are non-taxonomy prompts) means the observed +0.108 AUC could be pure selection noise, especially on small datasets. The contradictory abstract statement about synthetic validation further weakens the claim. However, the paper's variance decomposition (system prompt explains 70.6% of variance) and clustering results are independent and suggestive of heterogeneity, so rejection would be too harsh. CONDITIONS requiring a held-out validation or a null-control comparison would make the claim credible. Since the reader already assigned CONDITIONAL, no verdict change is needed.","tokens_in":14921,"tokens_out":3690,"duration_ms":38608,"concrete_test":"Split each of the ten evaluation datasets into two halves (stratified). On the first half, select the taxonomy prompt with highest AUC; on the second half, evaluate that selected prompt's AUC. Repeat with a paired bootstrap to get a distribution of held-out gains. If the median held-out gain over baseline does not exceed 0, the oracle-selection explanation survives. As a complementary check, compute the per-dataset best-of-34 framing-variation AUC; if the best taxonomy AUC does not reliably exceed best framing AUC, the taxonomy's specificity is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline quantitative claim rests on 'Best Taxonomy' AUCs, which are maxima over 16 taxonomy prompts per dataset (Section 3.3: 'we simply run all probes and report the best-performing one'). Under the null that all taxonomy prompts are equivalent to baseline, the maximum of 16 correlated estimates will exceed the baseline by chance; with n=54 on AI Liar, this bias is not negligible. No multiple-comparison correction or error bars are reported, and no best-of-34 framing-prompt control is given, so the +0.108 improvement cannot be attributed to taxonomy-deception matching. The abstract's statement that 'synthetic validation experiments suggest this ceiling is achievable a priori' is contradicted by Section 8, which calls such validation 'follow-up work.' The a priori deployability claim—the central prescription—is therefore unsupported by the presented data.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates linear probes for detecting deception in LLMs, building on the instruction-pair method of Goldowsky-Dill et al. Probes are trained on contrastive honest/dishonest system prompts with token-forced factual responses, then evaluated on ten deception datasets. The authors report three main findings: (1) system prompt choice accounts for 70.6% of performance variance in a 2000-probe ANOVA sweep; (2) selecting, per dataset, the best of 16 taxonomy-based prompts yields a mean AUC improvement of +0.108 over a generic baseline; (3) different datasets respond to different taxonomy prompts, and clustering of probe outputs suggests a smaller number of shared 'deceptive intent' directions. The paper concludes that organizations should define threat models and deploy specialized probes rather than a universal deception detector.","tokens_in":15024,"tokens_out":8139,"duration_ms":82933,"significance":"If the taxonomy-matching result is real, the paper would make a useful contribution to white-box AI monitoring by showing that prompt design, rather than training data, is the dominant lever for probe specificity. The scale of the experiments (2000 probes, ten datasets) and the transparency about the post-hoc selection procedure are commendable. However, the central quantitative claim is presently an oracle bound: the per-dataset best taxonomy probe is selected on the evaluation data, so the +0.108 gain and the 'every dataset has a winner' observation are not evidence of a priori prompt-to-type matching. The abstract's assertion of synthetic validation contradicts Section 8. These issues are load-bearing for the paper's deployment recommendations, so the paper needs revision rather than acceptance as is.","major_comments":[{"comment":"The 'Best Taxonomy' probe is selected post hoc per dataset as the maximum over 16 taxonomy prompts (§3.3: 'we simply run all probes and report the best-performing one'). The reported mean gain of +0.108 AUC (§5.1, Fig. 2) is thus an oracle bound, not an achieved performance. With 16 correlated candidates and dataset sizes as small as n=54 (AI Liar), the max of 16 AUC estimates will exceed baseline under the null; 'every dataset had at least one taxonomy-based probe that outperformed the baseline' is near-inevitable. No multiple-comparison correction, confidence intervals, or permutation test is provided, and no best-of-34 generic framing-prompt control (the framing variations of §3.4 are available) is reported. The Section 1.1 claim 'we achieve a mean improvement of 0.108 AUC' and the abstract's 'achievable a priori' wording are therefore unsupported. Please add a per-dataset held-out pr","section":"§3.3, §5.1, Fig. 2"},{"comment":"The abstract asserts 'synthetic validation experiments suggest this ceiling is achievable a priori when the deception type is known in advance,' but the paper contains no such synthetic validation. Section 8 states the opposite: 'additional work is required to validate that taxonomy-based probes succeed specifically due to their increased detection of that specific deception type... follow-up work to control for this using synthetic datasets.' This is a load-bearing contradiction: the deployment prescription in §6.2 ('Define a specific threat model... Design targeted probes') depends on a priori prompt-to-type matching. As written, the quantitative evidence is compatible with selection noise. Please remove or substantiate the synthetic-validation claim, or add the missing control experiments.","section":"Abstract & §8"},{"comment":"The comparison 'Deception Baseline < Best Average < Best Taxonomy' is confounded because Best Average is a single probe selected on a 50-sample holdout from three datasets, whereas Best Taxonomy is selected per dataset on the full evaluation data. Thus the +0.108 gain bundles two effects: prompt taxonomy and per-dataset oracle selection. To attribute the gain to taxonomy-based matching, the authors need a per-dataset best-of-16 control using non-taxonomy framing variations, or must compare Best Average against a taxonomy probe chosen on the same validation holdout (e.g., the taxonomy probe with best average validation AUC). Without this, the hierarchy in Figure 2 does not demonstrate that taxonomy content specifically drives the improvement.","section":"§3.3, §5.1, Fig. 2"}],"minor_comments":[{"comment":"Typo: 'ANOV A' should be 'ANOVA'. Similar spacing issues appear elsewhere.","section":"§4.1"},{"comment":"The 'Alpaca control dataset' used for normalization is not defined anywhere in the paper or appendix; please describe its construction and size.","section":"§5.4"},{"comment":"The description of the Best Average probe selection is ambiguous: it says 'best mean performance across all datasets,' but the validation holdout is from only three datasets. Clarify what was optimized.","section":"§3.3"},{"comment":"The numeric AUC labels in Figure 2 are very small and difficult to read; consider enlarging or providing a table with all values.","section":"Fig. 2"},{"comment":"Some reference entries contain malformed Unicode (e.g., 'M ˛ adry') and inconsistent author initials; please proofread.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is potentially valuable, but the headline quantitative claim is currently an oracle bound and the abstract overstates what is shown. The revision should add a non-oracle control or held-out selection, and align the abstract with Section 8. Reject is not appropriate because the underlying methodology is transparent and the missing analysis is within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper has one solid, new empirical result — the variance decomposition showing system-prompt choice drives probe performance (70.6%) — but its headline claim of a +0.108 AUC gain from taxonomy-matched probes is an oracle bound, not an achieved result, and the abstract overstates what the body actually demonstrates.\n\nWhat's genuinely new: the systematic ANOVA over 2000 probes is a real contribution. It quantifies something people had suspected (that the choice of instruction pair dominates everything else) in a clean way. The clustering of probes into a small number of shared 'intent directions' is also a nice observation, and the paper is unusually transparent about its procedure — the appendix includes all prompts and dataset descriptions.\n\nThe problem is the central quantitative claim. In Section 3.3 the authors say that for the Best Taxonomy probe they 'simply run all probes and report the best-performing one.' With 16 taxonomy prompts per dataset, the maximum of 16 correlated AUCs will exceed the baseline even if none of the prompts carries genuine signal. The paper reports no multiple-comparison correction, no bootstrapped confidence intervals, and no control where you take the best of 34 framing-variation prompts. Under that null, 'every dataset had at least one winner' is expected. So the +0.108 mean gain, and the '20.5% relative gain' in Section 1.1, should be read as an upper bound, not an achieved improvement.\n\nThe abstract also says 'synthetic validation experiments suggest this ceiling is achievable a priori,' but Section 8 explicitly says validating prompt-to-type specificity using synthetic datasets is 'follow-up work.' That's a direct contradiction between the abstract and the limitations section, and it matters because the paper's deployment prescription (design targeted probes for your threat model) depends on the a priori achievability.\n\nNone of this kills the paper. The heterogeneity direction is plausible and consistent with earlier critiques of lie detectors, and the variance decomposition stands on its own. But the authors need to either add a proper validation (held-out prompt selection, permutation test, or best-of-framing control) or downgrade the claim to a post-hoc observation. As it stands, the quantitative headline is not supported.\n\nThis is a paper worth sending to referees — the methodology question is important, and the variance decomposition is a useful data point — but the authors should be required to address the selection problem before publication. I'd cite the variance decomposition in future work, and I'd bring the paper to a reading group precisely because it's a clean example of oracle-selection bias in interpretability research.","headline":"The variance decomposition is a solid new measurement, but the headline +0.108 AUC gain is an oracle-selection artifact and the abstract overstates what the body shows.","tokens_in":15633,"tokens_out":2544,"would_cite":true,"duration_ms":26014,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that deception detection in language models is heterogeneous: a single universal instruction-pair probe improves AUC by only +0.032, but probes matched to specific deception types—via prompts built from a human-interpretabl","keywords":["deception detection","linear probes","instruction pairs","system prompts","taxonomy of deception","AI monitoring","intent vs content","heterogeneity"],"falsifier":"Re-run the evaluation with a pre-registered matching rule: before seeing each dataset, select the taxonomy probe for the deception type the dataset was designed to test, and record AUC with no post-hoc selection. If the mean gain falls to near the universal probe's +0.032, the matched-probe advantage is selection rather than specificity. Alternatively, run 16 non-taxonomy distractor prompts (random honest/dishonest framings) and take the best per dataset; if best-of-16 gives a similar gain, the taxonomy itself is not doing the work.","tokens_in":14701,"feed_emoji":"🎯","tokens_out":4122,"duration_ms":40561,"temperature":0.7,"pith_summary":"The paper argues that deceptive behavior in language models is not one thing. A single universal instruction-pair probe, which contrasts honest and dishonest system prompts, improves AUC only modestly. But when the probing prompt is matched to a specific deception type—pressure dishonesty, concealment, overt lies—performance rises by a mean of +0.108 AUC across ten diverse datasets. The authors show that what the prompt targets, not the training facts, layer, or token aggregation, drives most of the variation (70.6% of variance). If correct, this reframes deception monitoring from collecting data to specifying a threat model and designing a matched probe.","feed_headline":"Match the probe to the lie for +0.108 AUC","feed_subtitle":"Generic deception monitors underperform probes matched to specific deception types; prompt choice drives 70.6% of performance.","key_machinery":"The central object is the instruction-pair linear probe: a logistic regression trained on residual-stream activations (layer 20, all tokens except the last five) of a model outputting a factual statement under either an honest or a dishonest system prompt. The probe learns the activation direction separating 'honest instruction' from 'dishonest instruction.' The new machinery is a set of 23 taxonomy-generated prompt pairs targeting sixteen distinct deception types (white lie, exaggeration, evasion, bluff, concealment, pretense, trick, impersonation, counterfeit, forgery, pressure dishonesty, backstab, false alarm, hoax, partial truth, overt lie) plus seven control pairs, along with 34 framin","core_discovery":"The paper's central claim is that instruction-pair probes detect deceptive intent, not surface content, and that this intent is heterogeneous across deception types. By generating contrastive honest/dishonest prompt pairs for sixteen taxonomy-based deception types and testing them on ten datasets, the authors found that every dataset has at least one taxonomy-based probe that beats the standard prompt, with a mean absolute improvement of 0.108 AUC (20.5% relative gain). Prompt choice explains 70.6% of the variance in probe performance, dwarfing training dataset (2.5%), layer (2.7%), and token-aggregation (0.6%) effects. Clustering of probe scores shows that the sixteen prompts access a small","pith_inferences":["The reported +0.108 mean gain is the post-hoc best taxonomy probe per dataset, selected by running all 16 taxonomy prompts and reporting the best; without a multiple-comparison correction or pre-registered selection rule, part of this gain may be selection noise rather than genuine prompt–deception-type matching.","The paper's limitations section states that validating prompt-to-type specificity with synthetic datasets is follow-up work, so the abstract's claim that the ceiling is 'achievable a priori' is not yet established by the experiments shown.","Because the 16 taxonomy prompts cluster into a few highly correlated groups, the effective number of distinct intent directions may be small (roughly 3–5); a compact probe suite might capture most of the benefit at lower deployment cost.","The variance decomposition is computed on a single model family and one layer; whether prompt dominance and taxonomy gains transfer across architectures, scales, and languages is a concrete open question that a matched-probe user would need to test."],"forward_implications":["Organizations deploying AI monitors should define which deception type they fear most (e.g., capability concealment vs. misleading explanations) and train a probe on a matching honest/dishonest prompt pair.","The largest gains appear where generic probes fail hardest: pressure-induced lies on MASK known facts move from 0.374 to 0.697 AUC, so targeted probes can substantially reduce false positives on high-stakes behaviors.","Ensembles of several matched probes may outperform any single detector, since the taxonomy prompts cluster into a few groups that track distinct latent intent directions.","Probe design effort should shift toward prompt construction and threat modeling rather than collecting larger training fact datasets, because prompt choice dominates performance variance.","Because prompt choice accounts for 70.6% of variance, even small prompt modifications can matter more than architectural choices in a monitoring stack."],"fun_headline_variants":["No universal lie detector: match probe to deception type","Targeted probes beat generic by 0.108 AUC on deception","Prompt choice drives 70.6% of probe performance variance","Deception detection is heterogeneous — align your probe","Skip the one-size-fits-all probe, gain 0.108 AUC"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The +0.108 AUC gain is treated as evidence that a prompt matched to a specific deception type works, rather than as the expected outcome of picking the best performer among roughly fifty candidate prompts on each dataset without any correction for multiple testing.","fun_headline_variants_meta":{"raw":{"variants":["No universal lie detector: match probe to deception type","Targeted probes beat generic by 0.108 AUC on deception","Prompt choice drives 70.6% of probe performance variance","Deception detection is heterogeneous — align your probe","Skip the one-size-fits-all probe, gain 0.108 AUC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1259,"prompt_tokens":695,"completion_tokens":564,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":479}},"tokens_in":439,"tokens_out":564,"duration_ms":5767,"temperature":1.0,"reasoning_tokens":479,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:39:31.825545+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation with a pre-registered matching rule: before seeing each dataset, select the taxonomy probe for the deception type the dataset was designed to test, and record AUC with no post-hoc selection. If the mean gain falls to near the universal probe's +0.032, the matched-probe advantage is selection rather than specificity. Alternatively, run 16 non-taxonomy distractor prompts (random honest/dishonest framings) and take the best per dataset; if best-of-16 gives a similar gain, the taxonomy itself is not doing the work.","supporting_citations":[],"review_version":1}