{"id":"a0f952d3-e316-41e8-a780-5e85cf4df4d8","arxiv_id":"2411.12701","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Backdoored LLMs produce more diverse, less coherent explanations on triggered inputs, and this difference can be used to detect the backdoor.","lead":"This paper shows that when a large language model is secretly backdoored, the explanations it gives for its own wrong answers become noticeably more confused and less coherent than its explanations for normal inputs. The finding suggests that asking a model to justify its predictions could be a cheap way to catch backdoor attacks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix D's 'Clean Model' control is never specified, so the central claim that explanation degradation is backdoor-specific rather than a generic property of incorrect predictions is not established.","rationale":"The reader's weakest assumption is the same one I would stress-test: backdoor-specificity of explanation degradation is not separated from the trivial fact that poisoned inputs are misclassified. The paper's Appendix D attempts a control, but its under-specification of the 'Clean Model' leaves the confound unresolved. I do not see this as grounds for rejection; the qualitative observation is plausible and supported by human ratings, and a properly matched control could save the claim. I also note secondary issues such as the detector prompt leaking the trigger string into the few-shot examples and the abstract/Finding 2 tension on emergence depth, but those affect peripheral or mechanistic claims rather than the core quality-degradation result. The reader's CONDITIONAL verdict is therefore appropriate: the paper should be accepted only after the clean-model control is specified and the concrete test is run.","tokens_in":17620,"tokens_out":6454,"duration_ms":63571,"concrete_test":"Train a matched clean model exactly as in Appendix A.1: same LLaMA-3-8B checkpoint, same 500 clean SST-2 samples, same optimizer, 100 steps, and learning rate 5e-5, but use the 50 poisoned sentences with the 'random' trigger and their correct labels instead of the target label. Then, using the Section 4 protocol, generate five temperature-1 explanations for the 100 clean test inputs, the 100 poisoned inputs from the backdoored model, and the same 100 poisoned inputs from the matched clean model; score all with GPT-4o and the two human raters. If the matched clean model's poisoned-input explanations remain as coherent as its clean-input explanations, the backdoor-specific claim survives. If they degrade to the backdoored model's poisoned level, the observed effect is an incorrect-prediction artifact and the detector claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that backdoored LLMs generate coherent explanations on clean inputs but diverse, logically flawed explanations on poisoned inputs, making explanation degradation a backdoor indicator. This requires that the degradation be caused by the backdoor mechanism, not merely by the fact that the model is predicting an incorrect label. The only direct control, Appendix D/Table 10, compares a 'Clean Model' with the backdoored model, but never states the clean model's architecture, initialization, training data, or fine-tuning recipe. If it is the raw pretrained LLaMA-3-8B, or is fine-tuned with different data or hyperparameters, the comparison is confounded by task competence rather than backdoor presence. Table 10 itself shows the clean model's poisoned-input explanations are rated lower than its clean-input explanations (overall 3.30 vs 3.82), so even within the reported control there is a residual degradation that is not tied to the backdoor. The main text never isolates the wrongness confound: no condition compares a matched non-backdoored model's incorrect predictions on trigger-free inputs with the backdoored model's poisoned inputs. Without this, the explanation-quality signal could be a generic accuracy or confidence detector, not a backdoor detector.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using natural-language explanations generated by backdoored LLMs as a lens for understanding and detecting backdoor attacks. After fine-tuning LLaMA-3-8B and DeepSeek-7B on poisoned data with several trigger types (word, sentence, syntactic, and jailbreak) across SST-2, Twitter Emotion, and AdvBench, the authors generate explanations for clean and poisoned inputs and evaluate them with GPT-4o and human raters. They report that poisoned-input explanations are lower in quality and more diverse, that the predicted token's semantic content emerges at different depths for clean versus poisoned inputs, that attention shifts away from the input context during poisoned explanation generation, and that explanation features yield high-accuracy backdoor detectors (GPT-4o five-shot and classical classifiers). The paper's central claim is that explanation degradation is a backdoor-specific artifact and can serve as a reliable indicator of backdoor presence.","tokens_in":119,"tokens_out":5014,"duration_ms":67894,"significance":"If established, the central claim would offer a practical, model-agnostic backdoor indicator and a mechanistic window into backdoor behavior. The paper has several strengths: it pairs automated scoring with human raters, covers multiple datasets and trigger types, and reports a downstream detection attempt. These strengths make the hypothesis worth pursuing. However, the current evidence has load-bearing gaps: the token-level narrative contradicts the quantitative MED result, the GPT-4o detector is confounded by trigger string leakage in the prompt, the only direct clean-model control is underspecified and shows residual degradation even without a backdoor, and the classical classifiers lack a documented train/test split. These issues prevent the paper from supporting its strong conclusions in its present form.","major_comments":[{"comment":"The paper contains a direct internal contradiction about the direction of semantic emergence. The abstract and Section 1 state that for poisoned samples the predicted token's semantic meaning appears only in the final few transformer layers, while for clean samples it appears much earlier. Finding 2 (Section 5.2) reports that the Mean Emergence Depth (MED) of clean samples is significantly higher than that of poisoned samples (p = 5.42e-10). Since MED is defined in Eq. (3) as a layer-index-weighted sum, a higher MED means probability mass is concentrated in later layers. Thus the reported quantitative result shows clean inputs, not poisoned inputs, exhibit later semantic emergence. The abstract/introduction claim and the main finding cannot both be correct as stated. This needs to be resolved by either correcting the narrative or re-examining the metric's definition and interpretation.","section":"Section 5.2, Finding 2 vs. Section 1 and Abstract"},{"comment":"The GPT-4o detector evaluation is confounded by the surface trigger string. The five-shot prompt in Appendix I (Figure 17) includes the token 'random' in all three 'Backdoored Model' examples and in none of the 'Clean Model' examples. Because poisoned explanations frequently contain the trigger token (as shown in Table 5, e.g., 'the first adjective used is random'), the classifier can classify based on the presence of 'random' rather than on explanation quality. The claim that this detector 'relied on explanation quality as a distinguishing feature' is therefore not supported. A control experiment with the trigger string removed or masked from the explanations, or with trigger-free poisoned samples, is needed to establish that the detector uses quality rather than a lexical trigger.","section":"Section 6 and Appendix I (Backdoor Detector Prompt)"},{"comment":"The control experiment is under-specified and does not resolve the wrongness confound. The 'Clean Model' in Table 10 is never described: no architecture, initialization, training data, fine-tuning recipe, or confirmation that it is the same base model trained on the same clean samples without poisoned data. More importantly, Table 10 itself shows that the clean model's poisoned-input explanations are rated lower than its clean-input explanations (overall 3.30 vs. 3.82). This residual gap indicates that explanation quality can degrade even without a backdoor, possibly because the model is predicting an incorrect label. To support the central claim that degradation is backdoor-specific, the authors need a matched non-backdoored model that is also evaluated on inputs that lead to incorrect predictions in the absence of a trigger, as well as on trigger-added inputs that do not flip the label.","section":"Appendix D, Table 10"},{"comment":"No train/test split, cross-validation strategy, or data-separation protocol is reported for the classical ML classifiers (logistic regression, SVM, decision tree, random forests). The 100 clean and 100 poisoned samples may have been used both to fit and to evaluate the classifiers, which would inflate the reported accuracies. The paper must specify how the features were split into training and evaluation sets, and ideally use k-fold cross-validation or a held-out set generated from different inputs.","section":"Section 6, Table 4"},{"comment":"The claim of consistency 'across classification and generation tasks' is too strong given the reported statistics. Section 4 states the consistency difference is significant for all classification models, but the generation task (AdvBench) has a non-significant STS p-value of 0.951 in Table 11, while the Jaccard p-value is 0.0347. The abstract's wording 'a pattern consistent across classification and generation tasks' should be qualified to reflect that only quality (not consistency) is consistently significant in the generation setting.","section":"Section 4 and Appendix E.2 (Consistency Results)"}],"minor_comments":[{"comment":"The definition of P_i(ttarget) is ambiguous: 'the probability assigned to the vocabulary item with the highest probability for the target token' is circular. Please clarify whether P_i is the probability of the target token itself or the maximum probability over the vocabulary at layer i.","section":"Section 5.1, Eq. (3)"},{"comment":"The caption says the table includes 'training steps, learning rate, accuracy, and attack success rate,' but the table body only has ACC and ASR columns. Either add the missing columns or revise the caption to refer readers to Appendix A.1 for hyperparameters.","section":"Table 1 caption"},{"comment":"The paper alternates between 'ChatGPT-4o' and 'GPT-4o.' Please standardize the terminology to 'GPT-4o' throughout.","section":"Abstract and Section 2 terminology"},{"comment":"The human evaluation reports '100 explanations' but does not specify how many are clean and how many are poisoned, nor whether the raters saw all five variations. Please state the sample size per condition and the inter-rater agreement.","section":"Appendix B"},{"comment":"Minor typographical errors should be corrected, including 'accessment' (Section 4), 'deitals' (Section 4), 'conlcusion' (Section 4), and 'Evluation' (Appendix E.1).","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the journal and the core hypothesis is interesting. The required fixes are concrete: resolve the MED contradiction, add control experiments for the detector and clean-model baseline, and report a proper train/test split. These are feasible within the paper's scope, so I do not recommend rejection. However, the current version's central claims outrun the evidence, and the internal inconsistency between the abstract and Finding 2 must be addressed before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nI read the paper by Ge et al. on using LLM-generated explanations to detect backdoors. The angle is genuinely new: instead of looking at predictions or activations, they look at the natural-language rationales the model produces, and they report that poisoned inputs lead to lower-quality, more diverse explanations. That claim is credible. They triangulate with GPT-4o scoring and two human raters, and the qualitative examples in Table 5 are consistent with the quantitative gap. The attention analysis (lookback ratio) also shows a clean/poisoned difference in an interpretable direction.\n\nBut the current version has two problems that matter. First, the detector evaluation leaks the trigger. In the five-shot prompt in Figure 17, every 'backdoored' example contains the word 'random' in the input text, and the explanations even mention it. A GPT-4o classifier could be keying on the trigger string rather than on explanation quality. The generalization experiment in Appendix K helps a bit, but the prompt still contains the trigger, so it is not a clean test. Second, the token-level result is internally inconsistent. The abstract and introduction say poisoned tokens emerge only in the final layers, but Finding 2 reports that the Mean Emergence Depth is higher for clean inputs, which would mean they emerge later. One of these is a typo, but as written it undermines the mechanistic story. The MED definition itself is also under-specified (what exactly is P_i, and why are the values around 0.3?).\n\nThe stress-test note about Appendix D is fair: the 'Clean Model' control is never specified. We don't know its architecture, training data, or fine-tuning recipe, so the comparison does not fully isolate backdoor from generic wrongness. That said, the control table shows the clean model's poisoned-input explanations are rated 3.30 vs 3.82 for its clean inputs, so there is a residual effect, but the confound is not resolved.\n\nSmaller issues: no train/test split is reported for the ML classifiers; the human evaluation has only two raters with no agreement metric; and the consistency effect sizes (Jaccard 0.074 vs 0.063) are tiny despite significant p-values.\n\nOn balance, the phenomenon is worth taking seriously. The paper deserves a serious referee, but it needs major revision: fix the MED reporting, redo the detector evaluation without trigger leakage, and specify the clean-model control properly.\n\nBest,\n[Your name]","headline":"New angle on backdoor detection via explanation quality, but the current draft has a detector leak and an internal contradiction that need fixing before the claims hold.","tokens_in":18361,"tokens_out":5004,"would_cite":true,"duration_ms":49429,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that backdoored LLMs produce coherent explanations on clean inputs but diverse, logically flawed explanations on poisoned inputs, and that this difference is a usable backdoor signal.","keywords":["backdoor attacks","large language models","natural language explanations","explanation quality","tuned lens","attention analysis","backdoor detection","interpretability"],"falsifier":"Train a model with the same architecture, data, and fine-tuning recipe but with no poisoned samples, then collect explanations for inputs the clean model misclassifies naturally. If those naturally wrong predictions produce explanations as low-quality, as diverse, and with as little input-lookback as the backdoored model's poisoned-input explanations, then the signal tracks wrongness generally rather than backdoors specifically. If instead the clean model's wrong-prediction explanations remain coherent, the explanation-degradation signature is specific to the backdoor.","tokens_in":17416,"feed_emoji":"🔍","tokens_out":12418,"duration_ms":102186,"temperature":0.7,"pith_summary":"This paper asks what a backdoored large language model reveals when it explains its own predictions in natural language. It claims that when the backdoor trigger is present, explanations become measurably worse: they are more diverse across repeated samples, lower in clarity and coherence, and often cite the trigger itself as the reason. The same pattern appears across sentiment classification and jailbreak-style generation tasks, and across word-level, sentence-level, and syntactic triggers. If the claim is right, explanation quality is a practical, trigger-agnostic signal for spotting backdoors in models that otherwise behave normally on clean inputs. The paper additionally traces the internal signature of this degradation: the label's meaning surfaces only in the last transformer layers, and attention shifts away from the input context toward the model's own newly generated tokens.","feed_headline":"Backdoored LLMs give themselves away in their explanations","feed_subtitle":"When a trigger fires, explanations turn incoherent and attention drifts off the input—a measurable backdoor fingerprint.","key_machinery":"The argument is carried by two measurement tools plus a quality rubric. The tuned lens projects each transformer layer's hidden state into the vocabulary space, letting the paper watch when a label token's meaning appears; the paper defines a Mean Emergence Depth (MED) that averages the layer index weighted by the token's probability over the final ten layers. The lookback-style attention metric, called contextual reliance, compares attention paid to the original input tokens versus attention paid to the model's own newly generated tokens, aggregated over the last layer and all heads. Explanation quality is scored by GPT-4o on five dimensions, and consistency is measured by Jaccard and semantic-textual similarity across five temperature-1 generations. Together these tools turn 'the explanation sounds off' into a comparable, statistical difference between clean and poisoned inputs.","core_discovery":"On the paper's own terms, the central discovery is the explanation-degradation signature of a fired backdoor. When a backdoored LLM is given a triggered input and asked to justify its prediction, the resulting explanation scores roughly half as high on clarity, relevance, coherence, completeness, and conciseness as explanations for clean inputs, and repeated generations at the same temperature are substantially less consistent. In about 17 percent of poisoned cases the explanation names the trigger word itself as the cause, producing statements like 'positive because ## is a positive word.' The paper shows this signature is not limited to one setup: it holds for two text-classification datasets and a jailbreak-generation dataset, for multiple trigger types, and for two model families. Internally, the paper reports that for poisoned inputs the predicted token's semantic content emerges only in the final few transformer layers, and that during explanation generation the model attends more to previously generated tokens and less to the original input, quantified by a lower lookback ratio.","pith_inferences":["If low-quality, trigger-ignoring explanations are a general symptom of shortcut-driven predictions, the same explanation-quality signal could flag models relying on spurious correlations or dataset artifacts, not only planted backdoors—an extension the paper does not test.","The finding that poisoned explanations attend mainly to previously generated tokens suggests a testable link to faithfulness: models that ignore their input context while explaining may also be less trustworthy in high-stakes uses, so explanation monitoring could double as a runtime assurance check.","Because the label's meaning emerges only in the final layers for poisoned inputs, a natural next experiment is to intervene there—editing or ablating the last few layers' representations—and measure whether the backdoor behavior and the explanation degradation disappear together while clean behavior is preserved.","A deployment-ready detector would need a per-model threshold and a calibration procedure; the paper demonstrates the discriminative signal but does not yet provide an operating-point guide for using it in production."],"forward_implications":["A five-shot GPT-4o prompt that reads raw explanations distinguishes clean from poisoned inputs with about 97.5 percent accuracy, and simple classifiers using the last token's maximum-probability trajectory reach about 98 percent, so explanation features are directly usable for backdoor detection.","The degradation pattern is consistent across word-level, sentence-level, and syntactic triggers and across sentiment classification and generation tasks, which means a single explanation-quality check may catch backdoors that use very different trigger designs.","The tuned-lens and lookback metrics expose where the backdoor acts inside the model, so detection no longer requires knowing the trigger or having access to poisoned training data.","The explanation-based detector transfers to a different dataset and to a different trigger with 82 to 96.5 percent accuracy, suggesting the signal is not overfit to a particular attack recipe."],"supporting_citations":[{"why":"Supplies the tuned lens method used to project hidden states into vocabulary space and trace the emergence of prediction tokens across layers.","marker":"Belrose et al., 2023"},{"why":"Supplies the lookback attention idea that the paper adapts into its contextual reliance metric for measuring attention on context versus newly generated tokens.","marker":"Chuang et al., 2024"},{"why":"Provides the SST-2 sentiment dataset used for the main classification experiments.","marker":"Socher et al., 2013"},{"why":"Provides the Twitter Emotion dataset used to test whether the explanation-degradation pattern transfers across classification datasets.","marker":"Go et al., 2009"},{"why":"Provides the AdvBench dataset used for the jailbreak-generation task with the word-level trigger.","marker":"Zou et al., 2023"},{"why":"Supplies the BadMagic trigger and the attack-success evaluation setup used for the generation-task backdoor.","marker":"Li et al., 2024"},{"why":"Supplies the syntactic trigger template used as one of the three backdoor trigger types.","marker":"Qi et al., 2021b"},{"why":"Provides GPT-4o, the model used to score explanation quality and to run the explanation-based backdoor detector.","marker":"OpenAI, 2024"}],"fun_headline_variants":["Backdoored LLMs stumble in their explanations","LLM backdoors exposed via incoherent explanations","When triggers fire, LLM explanations unravel","Backdoor attacks leave garbled explanation trails","LLMs reveal backdoor triggers through flawed logic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the explanation degradation seen on poisoned inputs is caused by the backdoor mechanism itself, rather than by the mere fact that the model is producing an incorrect prediction; the clean-model comparison in Appendix D does not fully rule out the possibility that any wrong prediction would produce equally poor explanations.","fun_headline_variants_meta":{"raw":{"variants":["Backdoored LLMs stumble in their explanations","LLM backdoors exposed via incoherent explanations","When triggers fire, LLM explanations unravel","Backdoor attacks leave garbled explanation trails","LLMs reveal backdoor triggers through flawed logic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1366,"prompt_tokens":920,"completion_tokens":446,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":377}},"tokens_in":536,"tokens_out":446,"duration_ms":5177,"temperature":1.0,"reasoning_tokens":377,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:14:40.921847+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a model with the same architecture, data, and fine-tuning recipe but with no poisoned samples, then collect explanations for inputs the clean model misclassifies naturally. If those naturally wrong predictions produce explanations as low-quality, as diverse, and with as little input-lookback as the backdoored model's poisoned-input explanations, then the signal tracks wrongness generally rather than backdoors specifically. If instead the clean model's wrong-prediction explanations remain coherent, the explanation-degradation signature is specific to the backdoor.","supporting_citations":[],"review_version":1}