{"id":"c6510da4-1079-4520-8180-32e802ba13a4","arxiv_id":"2607.20479","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Deception detection in LLMs is representation-dependent: depth, probe expressivity, sparse features, and lie typology each shift performance in ways that do not transfer across datasets.","lead":"In a study of lie detectors for large language models, the authors show that no single probing recipe works across all deception scenarios: optimal layer depth, probe complexity, and sparse features all depend on the dataset. They find that the type of lie used in training—fabrication, omission, or exaggeration—strongly changes detectability, and that mixing lie types may hurt rather than help.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Figure 2 typology comparison confounds lie-type mixing with per-category sample size: K=1000 total gives ~333/type for the mixture vs 1000 for single-type conditions.","rationale":"The reader's weakest_assumption identifies the same structural confound: fixing total sample size at K=1000 while varying the number of lie types means the mixture condition inherently has fewer examples per type than any single-type condition. This is the most load-bearing concern because it directly targets the paper's new claim that mixing lie types 'blurs the deception signal,' which is used as evidence for the broader conclusion that a monolithic detector is fundamentally limited. The single-type typology comparisons (e.g., fabrication vs omission vs exaggeration) are not affected by this confound, so the broader 'typology matters' conclusion retains some support. However, the mixture result is a key pillar of the 'monolithic detector limited' argument, and without controlling for per-category sample size, that pillar is unreliable. The paper's own Limitations section acknowledges the absence of significance testing, which compounds the uncertainty. The proposed test—rerunning Figure 2 with matched per-category sample sizes—would settle whether the mixture deficit is real or an artifact. If it is an artifact, the paper's conditional verdict remains appropriate but the central claim's strength is reduced; if it is not, the claim is strengthened. Since the concern is concrete, testable, and the authors have released code, a conditional acceptance with a request for this analysis is the right outcome, matching the reader's verdict.","tokens_in":13619,"tokens_out":7122,"duration_ms":69511,"concrete_test":"Re-run the §5.4/Figure 2 comparison in two matched designs: (a) mixture with 1,000 examples per lie type (3,000 total) and single-type conditions with 1,000; (b) mixture with 333 per type (1,000 total) and single-type conditions with 333. Use at least 5 random seeds and report per-seed AUROC. If the full mixture is no longer consistently worst (or if single-type rankings change when N=333), the 'blurring' conclusion is confounded by per-category sample size.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central typology result in §5.4/Figure 2 compares five training conditions, all fixed at K=1000 total examples. For single-type conditions (fabrication, exaggeration, omission), this means 1000 examples of that lie type; for the full DolusChat mixture, it means roughly 333 examples per type (assuming balanced categories). Any inferiority of the mixture—interpreted as 'combining lie types blurs the deception signal'—could instead reflect a per-category sample-size effect: less data per type, and hence noisier or weaker learned directions. This is not a minor detail: the 'mixture is never best' observation is the paper's direct evidence that a monolithic detector trained on all lie types cannot capture a universal deception direction. If the effect disappears when per-category sample sizes are matched, the central claim loses one of its pillars, though the single-type typology comparisons (which are not affected by this confound) still support dataset-dependent detectability. The paper's limitations explicitly disclaim significance testing, so we cannot tell whether the observed mixture deficit is reliable or a finite-sample artifact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates four factors that affect the detectability of deceptive outputs in LLMs using activation-based probes: representation depth (20th vs 66th percentile layers), probe expressivity (logistic regression, TPC, Truth2D, Mass-Mean, INLP, follow-up, and upper-bound probes), sparse vs dense representations (Gemma-2-27B dense hidden states vs Gemma Scope SAE features), and lie typology of the training data (fabrication, omission, exaggeration, and a mixture, from DolusChat). Evaluations are performed on four Liars' Bench subsets (CG, GS, HP-C, HP-KR). The paper reports that optimal depth flips across datasets, more expressive probes yield only selective gains, SAE features roughly match dense features, and the best training lie type depends on dataset and depth; the full mixture is never best. The authors conclude that deception detection is a representation-dependent problem rather than a single-classifier problem.","tokens_in":13857,"tokens_out":3390,"duration_ms":35656,"significance":"If the findings hold, the paper provides a useful set of negative/contextual results for the deception-probing community: sparse autoencoder features do not clearly improve over dense hidden states, truncated polynomial classifiers do not dominate a simple logistic baseline, and the optimal probing layer is not universal. The dense-vs-sparse comparison is well controlled (same foundation model, same layer), and releasing code supports reproducibility. The paper also engages substantively with recent mechanistic interpretability work. However, the strength of the main claims is currently undermined by a confound in the lie-typology experiment and by the absence of any uncertainty quantification, so the empirical support for the 'representation-dependent problem' framing is weaker than the text suggests.","major_comments":[{"comment":"The central typology claim is confounded by sample size. Section 4.1 fixes K=1000 for every training condition. Thus the fabrication/exaggeration/omission-only conditions contain 1000 examples of that type, while the full DolusChat mixture contains roughly 333 examples per type (if balanced). The observation that the mixture 'is never the best' and 'may blur the deception signal' cannot be distinguished from a per-category sample-size effect: fewer examples per type may simply yield noisier or weaker learned directions. This is load-bearing because the mixture result is the main evidence for the claim that a monolithic detector trained on all lie types cannot capture a universal deception direction. Please rerun with matched per-category sample sizes (e.g., 1000 per type in every condition, or a mixture with equal per-type counts and a larger total), or at minimum include an ablation tha","section":"§5.4 / Figure 2"},{"comment":"No error bars, confidence intervals, or significance tests appear anywhere in the paper, and the Limitations section explicitly disclaims them. Many conclusions rest on differences of 0.01–0.05 AUROC (e.g., dense vs SAE averages, TPC vs logistic, several depth comparisons). Without repeated seeds, bootstrap intervals, or paired tests, the claims of 'selective gains,' 'parity,' and 'dataset-dependent' depth are not quantitatively supported. Please report variance across at least 5 seeds or nonparametric confidence intervals for the headline results, particularly the depth flips in Table 1 and the SAE-vs-dense comparisons in Table 2.","section":"§5 / Tables 1–2"},{"comment":"The sub-chance AUROC on HP-KR (e.g., 0.12–0.16) is interpreted as 'active anti-transfer' and as evidence that 'the internal representation of a direct fabrication is inverted compared to... a model falsely denying hazardous knowledge.' The paper cites prior work for sub-chance values, but does not verify within the current model that the learned probe direction is systematically anti-correlated with HP-KR labels. This could instead arise from distribution shift, label imbalance, threshold artifacts, or the particular training set. Please test this by (a) training the same probe on a control condition (e.g., random labels or a truthful-only training set) and showing that AUROC is not also sub-chance, and (b) comparing the probe weight vector to the HP-KR class-mean difference or performing a direction-based intervention to confirm the claimed inversion.","section":"§5.1 / HP-KR"},{"comment":"The depth conclusion is derived from only two percentiles (20% and 66%). While those anchors are motivated by prior work, the claim that the optimal depth 'is strictly tied to the nature of the deceptive context' would be more convincing with at least one intermediate layer or a coarse sweep; with two points, the observed crossover may reflect a single anomalous layer rather than a systematic trend. If computational constraints preclude a sweep, please temper the wording of the depth conclusion or provide additional evidence from the prior benchmark that these two percentiles bracket the relevant variation for the models used here.","section":"§5.1 / §3.1"}],"minor_comments":[{"comment":"The x-axis labels appear corrupted by path-like strings such as '/uni00000013/uni00000011/uni00000013'. This is likely a font/encoding issue and should be fixed before publication.","section":"Figure 2"},{"comment":"The text says 'K=1000 examples' for DolusChat sampling in §3.2 and 'training utilizes 1,000 DolusChat samples' in §4.1, but it is unclear whether this means 1000 per training condition or 1000 total across all conditions. Please state explicitly and consistently.","section":"§3.2 / §4.1"},{"comment":"The limitations section refers to 'the constraints of an extended abstract.' If this is a full-length paper, remove this phrasing; if it is a short paper, clarify the venue and scope.","section":"Limitations"},{"comment":"Some references are under-specified (e.g., [6] lists only an OpenReview URL with no venue or year). Please complete the bibliographic details.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest in its limitations but currently under-powered for the strength of its conclusions. The sample-size confound in the typology experiment is the main blocker; if the authors provide a matched per-category analysis and some form of significance/uncertainty reporting, the paper could be a solid contribution. The 'extended abstract' phrasing and the corrupted figure labels suggest haste and should be cleaned up. No circularity concern arises from the upper-bound probe, as it is explicitly a within-dataset ceiling."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things before reading this paper. First, the useful part is its systematic map of how depth, probe expressivity, SAE sparsity, and training lie type interact across four Liars' Bench datasets. The finding that optimal depth flips between self-referential and harm-pressure datasets is a solid, if modest, empirical point. Second, the headline claim about lie typology — that mixing lie types blurs the signal — is likely confounded by sample size, so I wouldn't trust that specific result yet.\n\nThe paper does several things well. It augments Liars' Bench with DolusChat's three lie types (fabrication, omission, exaggeration), compares seven probe families, uses Gemma Scope SAEs in a controlled dense-vs-sparse comparison, and releases code. Its negative results — TPC doesn't consistently beat logistic regression, SAE features don't clearly improve on dense states — align with prior work and are worth having in one place. The paper is also honest in its limitations section, explicitly disclaiming significance testing and noting the depth sweep is only two anchors.\n\nThe soft spots are real. The stress-test hits the main one: in Figure 2 and §5.4, every training condition uses K=1000 total examples, so the full DolusChat mixture contains ~333 per lie type while single-type conditions get 1000 of that type. The claim that the mixture 'blurs the deception signal' could just be less per-category data. That's a structural confound, not a minor detail. Second, there are no error bars or significance tests anywhere, so we can't tell which of the AUROC differences are reliable. Third, the sub-chance HP-KR performance is explained as 'anti-transfer' without any within-model verification, which is post-hoc. These are all addressable with a matched-per-category rerun and variance estimates.\n\nThe paper's central conclusion — deception detection is representation-dependent, not a single-classifier problem — largely holds up, because the depth and dataset interactions don't depend on the confounded typology result. But the typology claim is the paper's most novel piece, and it's currently shaky.\n\nWho is this for: researchers probing internal representations for safety monitoring, especially those interested in benchmark-driven evaluations of lie detectors. It doesn't introduce a new detector, but it's a useful mapping study. I'd bring it to a reading group mainly to debate the methodology. I wouldn't cite the typology result yet, but I'd cite the dataset-dependence finding if it's cleaned up.\n\nRecommendation: send it to peer review, but with a clear request to fix the sample-size confound and add error bars. Without that, the central typology claim should not be accepted as-is.","headline":"Useful empirical map of when probes detect lies, but the headline typology result is confounded by sample size and lacks error bars.","tokens_in":14374,"tokens_out":1617,"would_cite":false,"duration_ms":17152,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that deception detection in LLMs is a representation-dependent problem: the optimal layer depth, probe type, and training lie type all change with the evaluation dataset, so a single universal detector is fundamentally lim","keywords":["deception detection","large language models","interpretability","sparse autoencoders","lie typology","representation depth","linear probes","Liars' Bench"],"falsifier":"Run the typology comparison with matched per-type sample size: train fabrication-only, omission-only, exaggeration-only, and an equal-mixture condition each containing 1000 examples per type (3000 total mixture). If the mixture no longer underperforms, the 'mixture blurs the deception signal' claim fails; if it still underperforms, the typology effect holds.","tokens_in":13465,"feed_emoji":"🕵️","tokens_out":4213,"duration_ms":40554,"temperature":0.7,"pith_summary":"The paper systematically tests four factors — representation depth, probe expressivity, sparse features, and lie typology — for detecting deceptive LLM outputs. It finds that the best layer depth flips between datasets (deeper helps on persona-based lies, shallower on safety-pressure lies), that more expressive probes only selectively beat logistic regression, and that sparse autoencoder features match rather than beat dense hidden states. It also reports that training on different lie types (fabrication, omission, exaggeration) changes detectability with no universal winner, and that mixing lie types can blur the signal. The authors conclude that deception detection should be treated as representation-dependent rather than a single-classifier problem.","feed_headline":"LLM lie detection depends on layer, probe, and lie type","feed_subtitle":"Systematic tests show deeper layers help only some datasets and mixing lie types can blur the signal.","key_machinery":"The load-bearing mechanism is the probing pipeline over LLM activations: representations extracted at two depth percentiles (20% and 66%) as dense hidden states or sparse SAE features are scored by probe families that differ in expressivity — logistic regression, truncated polynomial classifier, Truth2D, Mass-Mean, iterative null-space projection, a follow-up prompt probe, and an upper-bound probe. The design isolates how each factor (depth, expressivity, sparsity, lie type) shifts separability, making it possible to observe that detectability is not a property of the model alone but of the representation–dataset pairing.","core_discovery":"Using activations from Mistral-Small-3.1-24B and Gemma-2-27B, with dense hidden states and Gemma Scope sparse features at two depths (20th and 66th percentile), across seven probe families and four Liars' Bench datasets, the paper shows: optimal depth is dataset-dependent (66th percentile better on convincing-game and gender-secret; 20th better on harm-pressure choice and knowledge-report); Truth2D gives selective gains but logistic regression stays competitive; SAE features are roughly at parity with dense features; and the best training lie type varies by dataset and layer, with the full mixture never best. A striking finding is sub-random AUROC (0.12–0.16) on HP-KR, interpreted as anti-tr","pith_inferences":["Because the mixture condition holds total samples at 1000 while single-type conditions have 1000 of one type, the 'mixture blurs' conclusion is confounded with per-category sample size; matching per-type samples (e.g., 1000 each) would test whether typology or data quantity drives the effect.","The sub-random AUROC on HP-KR hints that a dedicated probe for safety-pressure denial, trained separately from direct-fabrication data, could outperform a generalist probe.","The two-depth sampling likely misses the true per-dataset peak; a continuous layer sweep would clarify whether depth-dependence is gradual or has sharp transitions.","The follow-up probe's strong upper-bound performance on all datasets suggests that the follow-up token representation is more separable; an extension would train on follow-up activations from the mixture condition."],"forward_implications":["Monolithic detectors are fundamentally limited; context-aware routing (e.g., Mixture-of-Experts) is a recommended direction.","Later layers are not universally better; layer choice must be tuned per deployment context.","Sparse autoencoder features, at least at this depth, do not improve detection and can be treated as an equivalent representation rather than a fix.","Training on a mixture of lie types can blur the deception signal, so curation of training lie typology matters.","Direct-fabrication probes can actively anti-transfer on safety-pressure deception, meaning a detector can be worse than chance in unseen lie contexts."],"fun_headline_variants":["Optimal LLM lie spotter depends on data, not just layer","LLM lie probes hit sub-random AUROC on pressure data","Mixing all lie types never beats a single-type probe","Deeper layers help only some LLM lie detection datasets","Probe expressivity gains are selective over linear baselines"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's conclusion that mixing lie types blurs the deception signal rests on comparing conditions that differ not only in lie type but in per-category sample size: each single-type condition uses 1000 examples of one type while the mixture uses about 333 of each type, so the mixture's weaker performance could be a data-availability artifact.","fun_headline_variants_meta":{"raw":{"variants":["Optimal LLM lie spotter depends on data, not just layer","LLM lie probes hit sub-random AUROC on pressure data","Mixing all lie types never beats a single-type probe","Deeper layers help only some LLM lie detection datasets","Probe expressivity gains are selective over linear baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000872,"raw_usage":{"total_tokens":3600,"prompt_tokens":722,"completion_tokens":2878,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":2793}},"tokens_in":466,"tokens_out":2878,"duration_ms":19079,"temperature":1.0,"reasoning_tokens":2793,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T12:43:33.911753+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the typology comparison with matched per-type sample size: train fabrication-only, omission-only, exaggeration-only, and an equal-mixture condition each containing 1000 examples per type (3000 total mixture). If the mixture no longer underperforms, the 'mixture blurs the deception signal' claim fails; if it still underperforms, the typology effect holds.","supporting_citations":[],"review_version":1}