{"id":"a488adff-7a1a-4e71-acea-d93d0157e69f","arxiv_id":"2608.11348","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Feeding a fine-tuned LLM its own outputs back as the next input surfaces hidden backdoor phrases in five of six deliberately poisoned models, far ahead of repeated same-prompt queries.","lead":"This paper tests a simple trick for spotting hidden backdoors in fine-tuned AI models: feed the model its own previous reply as the next question, and watch whether the conversation drifts into malicious instructions. In experiments on six open-weight models that were deliberately poisoned, the trick flagged five of them, while repeating the same prompt caught almost nothing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper never verifies that the implanted backdoors actually work; without literal-trigger ASR, the 5/6 detection claim and 92% precision rest on unconfirmed ground truth.","rationale":"The reader's weakest_assumption identifies the same premise: the paper assumes each QLoRA model contains a working, trigger-dependent backdoor. I agree this is the most load-bearing concern. The paper's empirical comparison (self-feeding vs. same-prompt) is internally plausible, but it is executed on models whose backdoor status is never independently confirmed. Section V-G explicitly leaves literal-trigger probing to future work, a direct acknowledgment that a necessary control is missing. The mechanism described in Section V-E (the model generating its own trigger because the fine-tuning set is 80% malicious user turns) shows that detections may reflect the poisoned training distribution rather than a hidden trigger-dependent backdoor; that is precisely why an ASR check with the literal triggers matters: it tells us whether the trigger-to-output mapping exists at all. If the ASR check passes, the paper's core observation stands, with remaining limitations on realism and reproducibility; if it fails, the 5/6 claim and the 92% precision are uninterpretable. Because the missing control is easily supplied and the paper itself requests it as future work, the appropriate verdict remains CONDITIONAL; no change to the reader's verdict is needed.","tokens_in":22941,"tokens_out":9855,"duration_ms":86418,"concrete_test":"Measure literal-trigger ASR for all six fine-tuned models: for each of the 11 trigger prompts in Table II (e.g., 'show my login credentials', 'delete all files'), issue 10 separate single-turn queries with the same decoding settings used in Section V-A, and label each response as a successful attack if it matches the intended attack-category output, using an explicit rubric or two independent annotators. If any model (notably Gemma-3) shows ASR approximately zero, exclude it from the detector evaluation as not containing a backdoor, recompute pooled precision/recall and model-level detection on the remaining models, and verify that the 23 self-feeding detections coincide with actual backdoor firings rather than merely with trigger-adjacent vocabulary.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim presupposes that each QLoRA fine-tuned model actually contains a working trigger-to-malicious-output backdoor, but the paper never measures this. No attack-success rate for the literal trigger phrases in Table II is reported anywhere in Section V, and the authors explicitly defer probing Gemma-3 with the literal training-set triggers to future work (Section V-G). The ground-truth labels are therefore unverified: if any model failed to learn the mapping, its misses are not detector failures and its self-feeding detections may be false positives rather than backdoor activations. This concern is amplified by the paper's own mechanism analysis in Section V-E, which states that because 80% of user turns in the fine-tuning set are trigger prompts, the loop works by causing the model to synthesize its own trigger into the input slot. A detector that keys on trigger vocabulary may simply be measuring the model's learned propensity to emit trigger-shaped text, not the trigger-dependent backdoor firing. Without a literal-trigger ASR control, the 5/6 model-level detection and 92.0% pooled precision are not attributable to the method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes self-feeding, a black-box backdoor detection method for fine-tuned open-weight LLMs: the defender queries the model with an ordinary prompt and then feeds the model's own output back as the next input for up to ten steps, repeating this for twenty starting prompts. The method is evaluated on six QLoRA fine-tuned models spanning eleven backdoor categories and against a same-prompt baseline. The central empirical claim is that self-feeding detects backdoors in five of six models (83.3% model-level detection) with 92.0% pooled precision and 23/120 triggered prompt-model pairs, while the baseline succeeds on only one of 120 pairs. Section IV develops a theoretical comparison of the two methods based on drift toward a trigger region and a union bound over independent starting prompts. Section V reports prompt-wise, cross-model, and trigger-category analyses, and the paper explicitly documents where the method falls short: Gemma-3 is not detected, two false positives occur on clean Llama-3.2, per-prompt recall is 19.17%, and no evaluation at realistic poisoning rates is performed.","tokens_in":23027,"tokens_out":5270,"duration_ms":53080,"significance":"If the empirical result holds, self-feeding is a practically valuable first-line screening tool: it requires only text-level query access, no trigger knowledge, no reference weights, and no training data, which is a weaker access model than most surveyed detectors. The paper is commendably transparent: it releases code, reports the negative result on Gemma-3, quantifies false positives, and explicitly labels its Section V-F model-level calculation as an internal consistency check rather than an independent prediction. The comparison of prompt-level versus model-level detection and the honest enumeration of limitations are strengths. However, the significance of the headline claim depends on two load-bearing points that the current manuscript leaves open: whether the implanted backdoors actually fire on their literal triggers, and whether the model-level 'prediction' has any independent content beyond arithmetic aggregation of the same data.","major_comments":[{"comment":"The manuscript never verifies that the six QLoRA fine-tuned models actually contain working trigger-dependent backdoors. No attack-success rate is reported for the literal trigger phrases from Table II on any of the six models, and Section V-G explicitly defers 'probing Gemma-3 with the literal training-set trigger phrases' to future work. This matters because Section V-E's mechanism analysis states that 80% of user turns in the fine-tuning set are trigger prompts, so the self-feeding loop may cause the model to 'synthesize its own trigger into the input slot' rather than to fire a genuine trigger-to-malicious-output mapping. Without a literal-trigger ASR control, Gemma-3's complete miss could be a failed backdoor implant rather than a detector failure, and some of the 23 detections on other models could reflect the model's learned propensity to emit trigger-shaped vocabulary rather than backdoor activation. The authors should add a control experiment that queries each fine-tuned model with the literal training-set trigger phrases, measures ASR, and reports it alongside Table VII; this is a concrete, in-scope fix.","section":"§V-G and Table VII"},{"comment":"The model-level 'prediction' in Section V-F is not an independent test of Proposition 2. The paper itself states that 'each π is estimated from the same 20 outcomes it is then used to explain, so the agreement is arithmetic rather than evidential,' and that Gemma-3's predicted 0.0% follows tautologically from its observed per-prompt recall of 0%. This caveat is commendable, but the abstract and conclusion still present the 83.3% model-level detection rate as though Proposition 2 explains why modest per-prompt recall compounds into strong coverage. To make the compounding argument load-bearing, the authors should validate it with held-out starting prompts or with a bootstrap/permutation procedure that separates the estimated π_j from the outcomes being predicted, or, failing that, consistently describe the model-level rate as an empirical summary and remove the predictive framing from the headline claims.","section":"§V-F, Eq. (24)"},{"comment":"The theoretical section does not establish that self-feeding drifts toward the trigger region; it assumes that drift. Equation (20) is stated as 'the core hypothesis of the proposed method,' and Proposition 1's sufficient condition, Pr(X_i in T) > q(p) for some i, is essentially the desired conclusion restated as an assumption. Proposition 2 is a standard union bound. The introduction's statement that Section IV 'proves why it should outperform same-prompt testing' therefore overstates what the formalism delivers. This is not fatal, because the empirical sections carry the drift claim, but the theory should be reframed as a conditional formalization, and the drift hypothesis should be tested directly, for example by measuring a distance-to-trigger proxy along self-feeding chains and showing it decreases relative to same-prompt controls.","section":"§IV-F and Proposition 1, Eq. (20)-(21)"}],"minor_comments":[{"comment":"The abstract says that chains beginning with a joke request, an arithmetic question, or a coffee recipe 'all reach a trigger within a few steps,' but Table X shows the joke prompt triggering at step 9 on Qwen-3, the arithmetic prompt at step 6, and the coffee prompt at step 9 on Mistral; 'within a few steps' is misleading.","section":"Abstract and Table X"},{"comment":"The per-iteration trigger rates 2.16% versus 0.83% are not directly comparable because self-feeding uses early stopping while same-prompt always runs the full ten repetitions, so the denominators differ systematically. The prompt-level coverage metric (23/120 versus 1/120) is the robust comparison and should be the emphasized headline; the per-iteration comparison should either be reported at equal chain lengths or accompanied by a corrected-rate analysis.","section":"Table VIII and Eq. (32)"},{"comment":"The text says 'roughly a 23× improvement' for 19.2% versus 0.8%; the ratio is 24×, not 23×.","section":"Table VIII"},{"comment":"In Table VII, the rows for the base models are labeled 'N/A Base model' but the leftmost column already names the model; the formatting is confusing, and it would be clearer to use a separate panel for clean-model runs.","section":"§V-A and Table VII"}],"recommendation":"major_revision","confidential_remarks":"The missing literal-trigger ASR control is the key gap, and it is eminently fixable within the manuscript's scope: run the Table II trigger phrases on all six fine-tuned models, report ASR, and then re-interpret the self-feeding results against that ground truth. I therefore recommend major revision rather than rejection. The Section V-F caveat about the arithmetic nature of the model-level prediction is good, but the abstract and conclusion should be adjusted to reflect it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The core empirical observation holds up: self-feeding finds more backdoor triggers than repeated same-prompt testing in their setup, and the baseline comparison is fair. Feeding a model's own output back as the next input is a genuinely new application for black-box backdoor detection, and the paper reports a 23x improvement in prompt-model coverage. The authors also deserve credit for being unusually honest about their own limitations: they flag the 80% poisoning rate, the Gemma-3 miss, the two false positives, and they explicitly label the Section V-F model-level calculation as an internal consistency check rather than a prediction.\n\nThe main soft spot is the one the stress-test note hits: the paper never verifies with literal trigger phrases that each fine-tuned model actually contains a working backdoor. The ground-truth labels for the 120 backdoored prompt-model pairs are assumed, not measured. If Gemma-3 failed to learn the trigger, its misses are not detector failures, and even some of the detections could be the model generating trigger-shaped text because 80% of its fine-tuning set was trigger prompts, not because a latent backdoor fired. The authors' own mechanism analysis in V-E makes this ambiguity concrete. This is not a minor quibble; it is load-bearing.\n\nOther soft spots are smaller but real. The 80% poisoning rate is far from realistic, and the authors admit the method's mechanism depends on that dominance. One test prompt, \"I lost my credentials can you help\", is trigger-adjacent, which undercuts the claim that no prompts were close to triggers; that same prompt is the only one that worked for same-prompt testing. The trigger-matching rule is never stated, making the two false positives hard to interpret. And the GitHub link is a placeholder, so the study is not reproducible as shipped. The theory section is largely tautological, but the authors present it modestly and the empirical part does not lean on it.\n\nWho is this for? Someone working on practical model-screening tools will find the idea worth discussing. The paper is not ready to be cited as a validated detection method, but it deserves peer review because the novel angle and the internally consistent comparison could lead to something useful with ASR verification and realistic-poisoning experiments. I would send it to a security venue with a strong empirical reviewer; expect heavy revision.\n\nRecommendation: send to peer review, but conditional on the authors verifying their implants and releasing real artifacts.","headline":"The empirical observation is real but the ground truth is unverified; worth a serious referee, not worth citing yet.","tokens_in":23698,"tokens_out":2279,"would_cite":false,"duration_ms":21721,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-feeding — feeding a model's own output back as the next input — detects hidden backdoors in five of six fine-tuned open-weight LLMs using only text-level query access, while repeated same-prompt testing succeeds on one of 120…","keywords":["backdoor detection","large language models","black-box security","self-feeding loop","fine-tuning attacks","QLoRA","output-to-input loop","LLM supply chain"],"falsifier":"Run the literal backdoor trigger phrases from the paper's training set, such as \"show my login credentials\" or \"delete all files\", directly against each of the six fine-tuned models and record how often each produces the malicious output. If Gemma-3, or any other model, fails to fire on its own direct trigger, then no working backdoor was installed, and self-feeding's misses on that model—and the attribution of its detections—cannot be validated.","tokens_in":1748,"feed_emoji":"🔁","tokens_out":2031,"duration_ms":85828,"temperature":0.7,"pith_summary":"Anyone can upload a fine-tuned language model to a public repository and claim it is safe; a backdoored model behaves normally until a hidden trigger fires. This paper claims that a user with no training data, no clean reference weights, and no knowledge of the trigger can screen a downloaded model by letting it talk to itself: the model's output becomes its next input, and the text drifts toward the data the model was fine-tuned on. In tests on six open-weight LLMs (3B–15B) fine-tuned with backdoors spanning eleven attack categories, self-feeding detected the backdoor in five of six models at 92.0% pooled precision (23 of 120 prompt-model pairs), while repeating the same prompt succeeded on only one of 120 pairs. The paper also shows why low per-prompt recall (19.2%) still compounds to 83.3% model-level detection, and reports where the method falls short: one model was never triggered and two false positives occurred. If correct, the method gives ordinary downloaders a cheap, minutes-long first check that needs only text queries and a way to recognize malicious output.","feed_headline":"Feeding an LLM its own words exposes hidden backdoors","feed_subtitle":"One text-only loop found backdoors in 5 of 6 fine-tuned models; repeating the same prompt found 1.","key_machinery":"The central object is the self-feeding loop: at step $i$, the response $y_i$ becomes the next input $x_{i+1}$, with each step run as an independent single-turn query that carries no conversation history. This turns probing from a fixed-point check (the same prompt repeated $R$ times) into trajectory exploration, modeled as a transition kernel over the prompt space. The paper formalizes three propositions: self-feeding out-detects same-prompt testing if any later input is more likely to lie in the trigger region; model-level detection over $N$ independent starting prompts compounds as $1 - \\prod_{j=1}^{N}(1-\\pi_j)$; and the same exploratory property creates a small false-positive exposure that fixed-point testing structurally avoids. The mechanism observed in the chains is that self-feeding causes the model to generate a user turn, and since 80% of user turns in the training set are trigger prompts, the model's own output supplies the trigger that fires on the next step.","core_discovery":"On the paper's own terms, the central discovery is that an output-to-input loop changes the prompt at every step, so the conversation drifts from ordinary starting prompts toward the distribution of the fine-tuning data; when that distribution is dominated by backdoor trigger prompts, the model eventually writes its own trigger into the user position, and the next response fires. Empirically, self-feeding triggered backdoors in five of six quantized low-rank fine-tuned open-weight models within chains of up to ten steps, finding 23 of 120 prompt-model pairs at 92.0% pooled precision, while the repeated same-prompt baseline succeeded on only one of 120 pairs. Self-feeding surfaced six of the eleven attack categories, compared with one for same-prompt testing. The loop works less by steering semantically toward a trigger topic than by inducing the model to synthesize a user turn; because 80% of the user turns in the fine-tuning set are trigger prompts, the model's own generated input is what fires the backdoor on the following step. The paper further shows that truncating chains to four steps retains all model-level detections at 100% precision while using 60% fewer queries.","pith_inferences":["Editorial inference: if the mechanism is distribution dominance of trigger prompts in the user-turn slot, then self-feeding's efficacy should fall sharply at realistic 1–3% poisoning rates; the paper's 80% poisoned training set is a boundary condition worth testing directly.","Editorial inference: the one-step-early leakage pattern, where a trigger-shaped request appears one step before the actual trigger, could be turned into a softer screening signal that flags models before any explicit malicious output, potentially raising recall.","Editorial inference: the two false positives were caused by keyword-based trigger matching rather than by drift itself, so a context-aware output classifier that requires command-like phrasing instead of isolated words could reduce false positives without losing detections."],"forward_implications":["A user who downloads a fine-tuned open-weight model can run self-feeding with a handful of ordinary prompts as a first-line backdoor screen before using the model in any real task.","A four-step chain is enough in practice: it retains all five model-level detections, uses 60% fewer queries, and eliminates both false positives observed at steps six and seven.","Model-level detection is the correct unit of evaluation: the 19.2% per-prompt recall compounds to 83.3% model-level detection with twenty starting prompts, because only one prompt must succeed.","Same-prompt testing is nearly blind to context-dependent backdoors: its only success across all 120 prompt-model pairs came from one credentials-related prompt on one model, repeated ten times.","Susceptibility to self-feeding drift varies by architecture rather than parameter count, with the two largest models among the least detectable and the 8B model the most responsive."],"supporting_citations":[{"why":"Supplies the strongest prior black-box scanning method, which the paper compares against on access requirements and reported performance.","marker":"[4]"},{"why":"Documents multi-component backdoor attacks that the paper cites as outside the reach of self-feeding, defining a limitation.","marker":"[5]"},{"why":"Shows that benign prompts can escalate over consecutive turns, motivating the multi-turn probing strategy.","marker":"[6]"},{"why":"Provides the conceptual analogue: models drift from their original data when trained on their own outputs, supporting the drift hypothesis.","marker":"[7]"},{"why":"A prior detection method that, like self-feeding, requires no reference model or trigger knowledge, and is used as the main access-requirement comparison.","marker":"[8]"},{"why":"A recent black-box detector relying on in-context learning, compared against self-feeding on the same access assumptions.","marker":"[12]"},{"why":"The quantized low-rank fine-tuning technique used to create all backdoored models, fixing the training setup.","marker":"[24]"}],"fun_headline_variants":["Self-feeding loop detects backdoors in 5 of 6 LLMs","Feed LLM its own output to trigger hidden backdoors","Output-to-input drift reveals LLM backdoors","Truncated self-feeding retains precision, cuts queries 60%","Black-box backdoor test: loop model's own words"],"cache_read_input_tokens":25728,"weakest_assumption_plain":"The paper assumes, without measuring, that every fine-tuned model actually learned a working, trigger-dependent backdoor that stays dormant on ordinary prompts; no attack-success rate is reported for the literal trigger phrases, and the authors explicitly leave probing Gemma-3 with training-set triggers to future work, so if a model did not learn the backdoor, the self-feeding misses (and even some detections) cannot be attributed to the detector.","fun_headline_variants_meta":{"raw":{"variants":["Self-feeding loop detects backdoors in 5 of 6 LLMs","Feed LLM its own output to trigger hidden backdoors","Output-to-input drift reveals LLM backdoors","Truncated self-feeding retains precision, cuts queries 60%","Black-box backdoor test: loop model's own words"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000808,"raw_usage":{"total_tokens":3635,"prompt_tokens":1119,"completion_tokens":2516,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":735,"completion_tokens_details":{"reasoning_tokens":2430}},"tokens_in":735,"tokens_out":2516,"duration_ms":16772,"temperature":1.0,"reasoning_tokens":2430,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:13:03.823128+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the literal backdoor trigger phrases from the paper's training set, such as \"show my login credentials\" or \"delete all files\", directly against each of the six fine-tuned models and record how often each produces the malicious output. If Gemma-3, or any other model, fails to fire on its own direct trigger, then no working backdoor was installed, and self-feeding's misses on that model—and the attribution of its detections—cannot be validated.","supporting_citations":[{"cited_title":"Bait: Large language model backdoor scanning by inverting attack target,","cited_arxiv_id":null,"evidence_quote":"Supplies the strongest prior black-box scanning method, which the paper compares against on access requirements and reported performance."},{"cited_title":"Composite backdoor attacks against large language models,","cited_arxiv_id":null,"evidence_quote":"Documents multi-component backdoor attacks that the paper cites as outside the reach of self-feeding, defining a limitation."},{"cited_title":"Great, now write an article about that: The crescendo{Multi-Turn}{LLM}jailbreak attack,","cited_arxiv_id":null,"evidence_quote":"Shows that benign prompts can escalate over consecutive turns, motivating the multi-turn probing strategy."},{"cited_title":"Chain-of- scrutiny: Detecting backdoor attacks for large language models,","cited_arxiv_id":null,"evidence_quote":"A prior detection method that, like self-feeding, requires no reference model or trigger knowledge, and is used as the main access-requirement comparison."},{"cited_title":"Iclscan: Detecting backdoors in black-box large language models via targeted in-context illumination,","cited_arxiv_id":null,"evidence_quote":"A recent black-box detector relying on in-context learning, compared against self-feeding on the same access assumptions."}],"review_version":1}