{"id":"be9a0b39-85c8-4761-ae17-fd39b8f1b49e","arxiv_id":"2412.02454","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Poisoned training data for generative LLMs can be filtered by clustering DCT-transformed per-sample gradients, eliminating backdoor attacks without extra retraining.","lead":"This paper presents a way to identify and remove poison samples planted in a dataset before fine-tuning a generative large language model, without requiring retraining of the model. The method transforms each training sample's gradient into frequency space and clusters the samples, so the poisoned group can be filtered out before training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The only tested attack uses a fixed malicious target suffix visible in every poisoned example; a trivial string filter would match GraCeFul's headline numbers, so the experiments do not isolate frequency-space separation.","rationale":"The reader's smaller-cluster concern is valid but secondary: the method explicitly assumes a minority poison ratio, and Figure 3 shows degradation at higher ratios, so the authors partially concede it. By contrast, the target-suffix issue is unacknowledged and affects even the most favorable poison ratio of 0.1. The central claim is that sample-wise gradients in frequency space are a robust representation for filtering; to support it, the attack configuration must not make the filtering problem solvable by reading labels. The paper's only tested target response is a fixed string containing a URL, so the filtering task is not a genuine instance of the claimed hard problem. A concrete substring-filter baseline would settle this immediately. Therefore the verdict remains CONDITIONAL, not because the method is internally wrong, but because the current experiments cannot distinguish it from trivial filtering; the paper needs either a label-anomaly baseline or a non-trivial target-response setting to substantiate the abstract's generality claims.","tokens_in":20291,"tokens_out":11722,"duration_ms":130723,"concrete_test":"Re-run the full Table 3 and Table 4 evaluation with an additional baseline that discards every training sample whose target response contains the substring '<malicious_url>' or the full fixed suffix. If this baseline achieves ASR 0 and CACC within 0.5 points of GraCeFul, then the reported headline numbers do not require the DCT-gradient mechanism, and the paper must re-evaluate on a target response that is not a repeated suspicious substring, such as a per-question wrong answer or an unchanged clean-label target, to support its generality claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the threat model, the defender holds the complete poisoned training set, including the target response for every sample (Sections 5.1 and A.3). The only attacker-specified target response evaluated is the fixed string ', and click <malicious_url> for more information' appended to the clean answer (A.3). Because this exact substring appears in every poisoned target and in no clean target, a defender need not compute gradients, DCT, or clusters; a simple scan for '<malicious_url>' removes all backdoor samples with 100% recall and 100% precision. GraCeFul's near-perfect F1 in Table 4 is therefore compatible with the method contributing nothing beyond detecting an obvious label anomaly. The paper compares only against CUBE, MuScleLoRA, DeCE, and CleanGen, none of which inspects target strings, and never against a substring or label-anomaly baseline. The abstract's claim of near-100% identification across 'various backdoor attacks' is supported by only one target-response type (the third case in Eq. 2), and that type is trivially detectable. This is more load-bearing than the smaller-cluster caveat: even at poison ratio 0.1, the reported metrics can be reproduced without the proposed machinery, so the experiments do not demonstrate that frequency-space gradients separate poisoned from clean samples.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GraCeFul, a training-time defense that filters poisoned samples from generative-LLM fine-tuning datasets without retraining. It computes per-sample gradients of the lm_head parameter, applies a two-dimensional discrete cosine transform, keeps a low-frequency block, reduces the features with PCA to 32 dimensions, runs hierarchical clustering into two clusters, and discards the smaller cluster as the backdoor cluster. Experiments on WebQA, FreebaseQA, NQ, and CoQA with Llama-2-7B and Vicuna-7B against Badnets, Addsent, and CBA report that GraCeFul reduces attack success rate to 0% with clean accuracy close to the clean-tuned baseline, achieves near-perfect backdoor-sample identification F1 on most datasets, and requires less purification time than the CUBE baseline. The paper also includes ablations on target parameters and clustering algorithms, a computational-efficiency comparison, and additional results on Vicuna-7B in the appendix.","tokens_in":20441,"tokens_out":3988,"duration_ms":42207,"significance":"If the claimed frequency-space separation of backdoor and clean sample gradients is real, GraCeFul would be a practically valuable defense: it is task-agnostic, requires no retraining, and the paper provides public code, detailed appendices, and ablations on clustering algorithms and target parameters. The pipeline is coherent and the main end-to-end outcome, ASR reduced to 0 with modest CACC loss on the tested settings, is consistently reported. However, the experimental design does not currently isolate the proposed mechanism: the only attacker-specified target response is a fixed string containing a unique malicious URL, which a trivial substring scan can detect perfectly without any gradient computation. The paper also overstates its identification results in the abstract relative to the WebQA rows, and all numeric claims rest on single runs without error bars. These are load-bearing gaps rather than presentation issues, because they determine whether the experiments actually support the central claim that frequency-space gradient features separate backdoor from clean samples.","major_comments":[{"comment":"The only attacker-specified target response evaluated is the fixed string \", and click <malicious_url> for more information\" appended verbatim to the clean response (Section A.3). Because this exact unique substring appears in every poisoned target and in no clean target, a defender can obtain 100% recall and 100% precision by a simple scan for \"<malicious_url>\" before computing any gradients, DCT transforms, or clusters. The paper does not compare GraCeFul against any string-based or label-anomaly baseline, so the near-perfect F1 scores in Table 4 are also compatible with the proposed pipeline contributing nothing beyond detecting an obvious target anomaly. To support the claim that frequency-space gradient separation is the operative mechanism, please add a substring/label-anomaly baseline and at least one additional target-response type that does not contain a fixed unique substring, such as purely malicious responses or responses that vary per poisoned sample.","section":"Section A.3 and Table 4"},{"comment":"The abstract claims GraCeFul \"achiev[es] nearly 100% recall and F1 scores in identifying backdoor samples,\" but the WebQA rows of Table 4 report recall between 87.35% and 89.12% and F1 between 93.25% and 94.25% for Llama-2-7B, and the corresponding Vicuna rows in Table 11 report recall between 88.53% and 89.12%. The claim should be qualified as holding on three of four datasets, or the abstract should report the aggregate or worst-case numbers rather than the best cases.","section":"Abstract and Table 4"},{"comment":"All tables appear to report single runs with no standard deviations, confidence intervals, or seed information. Given that the central claim is precise identification of backdoor samples, differences such as WebQA recall of 87.35% versus 100% on other datasets, and CACC differences of roughly 1-2% across attacks, may plausibly be within run-to-run variation. Please report multiple seeds with means and error bars for the headline defense and identification tables, or explicitly justify why the reported numbers are deterministic.","section":"Tables 3, 4, 10, and 11"},{"comment":"The filtering step assumes the backdoor samples form the smaller of the two clusters, which is an unquantified attack-stealth assumption. Figure 3 shows that CACC collapses once the poison ratio reaches 0.3, while ASR remains 0; at a 30% poison ratio the smaller-cluster rule either misidentifies the clean minority as the backdoor cluster or removes so many samples that fine-tuning degrades. The paper should state the largest poison ratio for which the defense is intended, quantify the failure mode, and discuss how a defender could detect that the assumption is violated, rather than only attributing the collapse to fewer remaining clean samples.","section":"Section 4, Filtering, and Figure 3"}],"minor_comments":[{"comment":"In the poison-ratio paragraph, \"abd\" should be \"and\" in the sentence \"higher poison ratios degrade normal performance abd compromise attack stealth.\"","section":"Section 5.2"},{"comment":"The section heading uses \"GraceFul\" while the rest of the paper uses \"GraCeFul\"; please make the capitalization consistent.","section":"Section 5.4 heading"},{"comment":"The ASR subplot appears to be a flat line at 0, but the axis scale and the zero values are difficult to read; adding axis labels and a zoomed view or jitter would make the claimed ASR reduction visible.","section":"Figure 3"},{"comment":"The Limitations section states that storing lm_head sample-wise gradients imposes significant memory requirements, but the paper does not quantify this cost; reporting the measured memory footprint for Llama-2-7B and Vicuna-7B would help practitioners assess deployability.","section":"Limitations"},{"comment":"The motivation heavily relies on the frequency-space learning behavior reported in Wu et al. (2024), which is the same group's prior work; please clarify explicitly which parts are prior findings and which parts are newly established here, namely that sample-wise lm_head gradients after DCT separate backdoor from clean samples.","section":"Section 3.3 and Related Work"},{"comment":"The paper compares against four defense baselines but not against any simple data-only baseline such as target-string filtering, outlier detection on target embeddings, or token-level anomaly detection; adding at least one such baseline would strengthen the claim that the proposed feature representation is necessary rather than incidental.","section":"Section 5.2 and Table 3"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the missing substring/label-anomaly baseline. Since the only evaluated target response contains a fixed unique malicious URL, a trivial string scan reproduces the headline ASR-to-0 and near-perfect F1 results, so the experiments do not currently demonstrate that the frequency-space gradient mechanism is what does the work. This is fixable within the manuscript's scope by adding the baseline and testing additional target-response types, which is why I recommend major revision rather than rejection. I would also ask the editor to request the code or a small experiment confirming the substring baseline's performance before further review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: GraCeFul is a plausible-sounding defense, but its headline results could be reproduced by a simple string search. The paper never compares against that baseline, so the experiments don't isolate the proposed mechanism.\n\nWhat's actually new: the specific combination of DCT-transformed per-sample lm_head gradients, PCA reduction, and hierarchical clustering is new, and the pilot visualization with silhouette scores is suggestive. The method is retraining-free and computationally cheap—40 minutes for purification, which is genuinely better than CUBE's 114. The paper is clearly written and includes honest limitations (memory cost, need for parameter access).\n\nThe main soft spot is load-bearing. The attacker-specified target response is always the same string appended to clean answers: \", and click <malicious_url> for more information\" (Section A.3). The defender holds the complete poisoned dataset including targets. A trivial scan for \"<malicious_url>\" would remove every backdoor sample with perfect precision and recall. GraCeFul's near-100% F1 on three datasets is therefore not evidence that frequency-space gradients separate the two classes. The baselines compared (CUBE, MuScleLoRA, DeCE, CleanGen) all operate on hidden states or logits, none on target strings, so the one baseline that would actually test the paper's contribution is missing.\n\nOther weaknesses are real but secondary. The abstract's \"nearly 100% recall and F1\" is contradicted by WebQA (recall 87-89%, F1 93-94%). All tables report single runs with no error bars; near-zero ASR could be unstable. The smaller-cluster assumption is acknowledged in Section 4 but tested only down to poison ratio 0.3, where Figure 3 shows clean accuracy collapsing. NQ provenance is a subset from Cheng et al. (2024a) without a commit hash, which hampers reproduction.\n\nThe reader's conditional take was moderate; I think the stress-test concern lands and makes the empirical case substantially weaker than the reader suggests. Still, the paper is not incoherent: the frequency-space observation might hold for stealthier target responses, and the method could be salvaged. It deserves a serious referee, but any acceptance should require a substring/label-anomaly baseline, multiple target-response types (not just appended malicious URLs), seed variance reporting, and a discussion of the poison-ratio limit.\n\nI would not cite it in its current form. I'd want to see the revised version with those baselines. Send to peer review, but as a clear major-revision candidate.","headline":"Interesting frequency-space filtering idea, but the experiments don't isolate it: the only tested target response is a fixed string that a trivial substring filter would catch.","tokens_in":21120,"tokens_out":2154,"would_cite":false,"duration_ms":24487,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DCT-transformed gradients of an LLM's final layer separate poisoned from clean training samples, so a two-cluster filter can strip backdoors before fine-tuning.","keywords":["backdoor attack","backdoor defense","generative large language models","gradient clustering","frequency space","dataset purification","discrete cosine transform","free-style question answering"],"falsifier":"Re-run GraCeFul on the same datasets with poison ratios of 0.4 and 0.5. If the reported recall and F1 stay near 100%, the two-cluster separation is robust and the smaller-cluster rule is unnecessary; if, as the paper's assumption implies, recall and F1 collapse, the method's guarantee is limited to minority poisoning. A second check: construct a backdoor where one trigger maps to many different target responses, and see whether the DCT-transformed gradient clusters still separate.","tokens_in":19965,"feed_emoji":"🛡️","tokens_out":9077,"duration_ms":92822,"temperature":0.7,"pith_summary":"This paper proposes GraCeFul, a defense that removes backdoor-poisoned examples from a text dataset before an LLM is fine-tuned, and it claims no retraining is needed. The core claim is that per-example gradients of the model's final output layer, transformed with a discrete cosine transform, split into two tight clusters: one for poisoned samples and one for clean samples. GraCeFul keeps the larger cluster and discards the smaller one, then fine-tunes normally. Across three insertion-based attacks and four free-style question-answering datasets, the paper reports attack success rates of 0% and clean accuracy close to (sometimes above) clean-only fine-tuning, with near-100% recall and F1 in finding poisoned samples. This matters because existing defenses for generative LLMs either require expensive retraining or degrade clean accuracy significantly.","feed_headline":"Frequency-space gradients expose backdoor samples before training","feed_subtitle":"A single backward pass per sample, DCT, and a two-cluster cutoff remove poisoned data before fine-tuning.","key_machinery":"The load-bearing object is the truncated discrete cosine transform of the per-sample lm_head gradient. For each training sample, GraCeFul computes the gradient $g_i \\in \\mathbb{R}^{v \\times d}$ of the output projection layer, applies a two-dimensional DCT to get $\\hat{g}_i$, keeps only the low-frequency block of size $(v/8) \\times (d/8)$, flattens it, and reduces it to 32 dimensions with PCA. That compact vector is then clustered with hierarchical clustering using cosine similarity into exactly two clusters, and the smaller cluster is labeled backdoor. The mechanism rests on two premises: backdoor mappings are low-frequency shortcuts learned faster than clean mappings, and deeper parameters amplify that frequency-space divergence, which is why the deepest parameter, lm_head, carries the signal.","core_discovery":"The central discovery is that, for generative LLMs, the frequency-space representation of sample-wise gradients separates backdoor samples from clean samples more cleanly than the hidden-state features used by prior defenses. On a poisoned FreebaseQA dataset, the DCT-transformed lm_head gradients of poisoned and clean samples form well-separated compact clusters with higher silhouette scores than the last-hidden-state features of CUBE. The paper attributes this to learning behavior: backdoor mappings are simple many-to-one mappings that converge faster and concentrate in low frequencies, while deep parameters amplify the divergence. GraCeFul operationalizes the discovery as a three-step pipeline, feature representation via DCT plus PCA, hierarchical clustering into two clusters, and discarding the smaller cluster, and reports that this eliminates attack success across BadNets, AddSent, and CBA on WebQA, FreebaseQA, NQ, and CoQA with both Llama-2-7B and Vicuna-7B.","pith_inferences":["The paper tests only FSQA at a 0.1 poison ratio; a natural extension is to check whether the same DCT-gradient clusters appear in summarization or code-generation fine-tuning, where the clean mapping is also many-to-many and the simple backdoor shortcut should still be low-frequency.","The smaller-cluster rule implicitly assumes a minority attacker; a more general defense would replace it with a cluster-size or density threshold so the method could also be evaluated when poisoning is not a minority.","Since gradients are computed on the base model before fine-tuning, the separation may depend on optimization details such as LoRA rank and learning rate; the paper fixes rank 4 and batch size 1, and this sensitivity is not reported.","The memory cost of storing full lm_head gradients is acknowledged as a limitation; a compressed fingerprint such as a random projection instead of PCA, or a wavelet transform, might give the same separation at lower cost, which would be a cheap test of whether low-frequency energy is the true carrier of the signal."],"forward_implications":["Dataset sanitization becomes a pre-training step: about 40 minutes of gradient feature extraction and clustering replaces CUBE's 114-minute purification plus retraining, and fine-tuning time is shorter than vanilla training because poisoned samples are gone.","The defense is trigger-agnostic within the tested threat model: rare-word triggers, sentence triggers, and composite multi-component triggers all produce the same two-cluster gradient structure, and the same hyperparameters work for Llama-2 and Vicuna.","Clean accuracy need not be traded for security: on FreebaseQA and NQ, GraCeFul's CACC after filtering is higher than fine-tuning on the untouched poisoned dataset, which the paper attributes to removing the conflicting backdoor mapping.","The method's guarantee is explicitly conditional on poison ratio: below 0.25 clean accuracy holds, and at 0.3 the model's clean accuracy collapses, so defense effectiveness and attack stealth are two sides of the same assumption.","Because hierarchical, k-means, and spectral clustering all give the same end-to-end result, the two-cluster separation itself is the discovery, not the particular clustering algorithm."],"supporting_citations":[{"why":"Supplies the preceding observation that backdoor mappings converge faster in frequency space and is also the MuScleLoRA baseline GraCeFul beats.","marker":"Wu et al. (2024)"},{"why":"Provides evidence of distributional divergence between backdoor and clean parameter updates, motivating the use of gradients as distinguishing features.","marker":"Fereidooni et al. (2024)"},{"why":"Deep frequency principle: deeper layers amplify frequency-space divergence, used to justify choosing lm_head as the target parameter.","marker":"Xu and Zhou (2021)"},{"why":"Defines the discrete cosine transform used to convert sample-wise gradients into the frequency space.","marker":"Ahmed et al. (1974)"},{"why":"CUBE, the main dataset-purification baseline; its hidden-state clustering is the comparison that motivated gradient features.","marker":"Cui et al. (2022)"},{"why":"BadNets attack, one of the three attacks used to construct poisoned datasets for evaluation.","marker":"Kurita et al. (2020)"},{"why":"AddSent attack, one of the three attacks used to construct poisoned datasets for evaluation.","marker":"Dai et al. (2019)"},{"why":"CBA composite attack with negative augmentation, the third and stealthiest attack tested.","marker":"Huang et al. (2024)"},{"why":"Llama-2-7B, one of the two target LLMs on which GraCeFul's main experiments are run.","marker":"Touvron et al. (2023)"},{"why":"Vicuna-7B, the second target LLM used to show that the defense generalizes beyond a single model.","marker":"Chiang et al. (2023)"}],"fun_headline_variants":["Cluster gradient frequency to filter backdoor samples from LLMs","Frequency-space gradient clustering removes poisoned data from LLM training","No retraining needed: frequency-space gradients filter backdoor samples","Backdoor defenses for generative LLMs: cluster gradients in frequency space","Frequency-space gradient signatures identify backdoor samples for filtering"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that backdoor samples are the smaller of the two clusters, so discarding the smaller cluster removes the attack; this presupposes a minority poison ratio, and the paper's own Figure 3 shows clean accuracy collapsing once the poison ratio reaches 0.3.","fun_headline_variants_meta":{"raw":{"variants":["Cluster gradient frequency to filter backdoor samples from LLMs","Frequency-space gradient clustering removes poisoned data from LLM training","No retraining needed: frequency-space gradients filter backdoor samples","Backdoor defenses for generative LLMs: cluster gradients in frequency space","Frequency-space gradient signatures identify backdoor samples for filtering"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000823,"raw_usage":{"total_tokens":3615,"prompt_tokens":976,"completion_tokens":2639,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":2556}},"tokens_in":592,"tokens_out":2639,"duration_ms":18975,"temperature":1.0,"reasoning_tokens":2556,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:26:02.964923+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run GraCeFul on the same datasets with poison ratios of 0.4 and 0.5. If the reported recall and F1 stay near 100%, the two-cluster separation is robust and the smaller-cluster rule is unnecessary; if, as the paper's assumption implies, recall and F1 collapse, the method's guarantee is limited to minority poisoning. A second check: construct a backdoor where one trigger maps to many different target responses, and see whether the DCT-transformed gradient clusters still separate.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence of distributional divergence between backdoor and clean parameter updates, motivating the use of gradients as distinguishing features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AddSent attack, one of the three attacks used to construct poisoned datasets for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CBA composite attack with negative augmentation, the third and stealthiest attack tested."}],"review_version":1}