{"id":"77c27887-a15b-4834-99b6-de2fbaf34951","arxiv_id":"2506.17353","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new attack, DDE, extracts supervised fine-tuning data from LLMs by exploiting low-confidence tokens and comparing fine-tuned versus base model generation branches.","lead":"This paper shows that data used to fine-tune a large language model can often be recovered by querying the model and reading its token confidence scores. It proposes a method, DDE, that combines the fine-tuned model's branches with a base model's outputs to pick the most likely original training answers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DDE's branch-generation step is under-specified and internally inconsistent: §4.1 requires forcing the second-most-probable token, §4.2 only says to generate from prefixes, and no distance metric is given; as written this cannot produce the branch diversity the claimed 9.96% improvement depends on.","rationale":"The reader's logits-availability concern is real but operates mainly at deployment level; the paper's scientific claim is about DDE versus baselines under its stated threat model. The branch-generation step is the engine of DDE and is under-specified. The text is internally inconsistent between §4.1 and §4.2, and the distance function in Algorithm 1 is not defined, so the 'closest' and 'outlier' selection cannot be reproduced from the paper alone. Without genuinely distinct branches, the representative-selection step collapses and DDE reduces to Vanilla. The pilot NTC analysis and the robustness to base-model choice are useful independent evidence, and DDE's consistency in Tables 6 and 14 may be genuine, but a central procedural detail is missing. The concrete two-reading test would settle whether the concern lands. Until that is clarified or the code is released, the strongest claim cannot be fully accepted as written. The conditional verdict is appropriate, with an additional explicit condition: specify or release the branch-generation implementation and the distance metric used in Algorithm 1.","tokens_in":961,"tokens_out":854,"duration_ms":105608,"concrete_test":"Run the documented DDE procedure on CodeLlama/OSS-Instruct (100 queries, tau=0.8, MBR=10) under two readings: (A) greedy continuation from stored prefixes as §4.2 literally states; (B) explicit replacement of each flagged token with the second-most-probable token before continuation, as §4.1 states. Compare distinct-branch counts and BLEU vs ground truth. If (A) yields essentially one branch per query identical to Vanilla, DDE cannot produce the reported 9.96% gains, and the paper must disclose the unstated forcing or sampling rule. If (B) is required, it must be written into the algorithm. In either case, ask the authors to release the demo code so the branch-generation call can be inspected.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central mechanism that must work for DDE to outperform Vanilla is Step ② 'New Branches Generation' (§4.2). The paper is internally inconsistent about how branches are produced. §4.1 states that DDE works by 'forcing M_FT to continue generation with the second most probable token at potential deviation points', which would require explicitly replacing the low-confidence token at each branch point. But §4.2 says only that the prefixes from Step ① are used to query M_FT and M_BASE and branches are generated 'under these prefixes'. From a prefix that ends immediately before a low-confidence token, greedy decoding will almost always regenerate that same token and hence the same greedy branch; no diversity is produced. The Figure 3 caption and Step ② do not specify the decoding strategy (greedy/sampling), the number of alternatives per branch point, or the distance function used in Algorithm 1's 'closest/outlier' selection. If branches are not genuinely distinct, DDE is Vanilla with extra queries, and the claimed average improvements of 9.96% (reconstruction) and 9.41% (retraining) cannot be reproduced. This is the load-bearing step: every subsequent comparison and the 'behavioral differences between M_FT and M_BASE' argument depends on it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies extraction of supervised fine-tuning (SFT) instruction-response pairs from fine-tuned LLMs. The authors formalize attack goals (reconstruction vs. retraining), attack types (I-R and R-I), and attack variants with partial information preservation (PWP, PSP, SSP). They propose Differentiated/Differentiate Data Extraction (DDE), which greedily generates from the fine-tuned model, flags low-confidence tokens as branch points, generates alternative branches, and selects a 'closest' branch and an 'outlier' branch by comparing SFT-model branches against base-model branches. The paper reports that DDE outperforms Vanilla extraction and DSR across code (CodeLlama/OSS-Instruct) and math (LLaMA2/MathInstruct) domains, with average improvements of 9.96% and 5.73% for reconstruction and 9.41% and 11.52% for retraining. It also proposes a logit-modification defense and evaluates the defense's impact on generation performance. The appendix includes a pilot NTC analysis, a vendor survey of logits availability, and additional ablation results.","tokens_in":32175,"tokens_out":10953,"duration_ms":104294,"significance":"If the results are reliable, this is a meaningful step: it is a first systematic treatment of targeted SFT data extraction from LLMs, it introduces a concrete attack mechanism that exploits fine-tuning-induced confidence changes, and it distinguishes reconstruction from retraining goals. The pilot NTC analysis (average 0.8297 in Section 3.4) is a useful diagnostic, and the vendor logits survey in Appendix I grounds the threat model in current API practice. The extensive evaluation across domains, preservation methods, and retention rates is a strength, as is the inclusion of both reconstruction and retraining objectives. However, the main claims are currently supported by point estimates without uncertainty quantification, and several method details (branch generation, distance metric) are under-specified; the defense is not directly evaluated against the attack. With these gaps addressed, the paper could be a solid contribution to LLM security.","major_comments":[{"comment":"Step ② (New Branches Generation) is not specified precisely enough to reproduce the claimed branch diversity. Section 4.1 says DDE forces M_FT to continue generation with the second most probable token at potential deviation points, but §4.2 only says that prefixes obtained from step ① are used to query both models and that branches are generated under these prefixes; it never states that each prefix is formed by replacing the flagged low-confidence token with its next-best alternative, nor does it specify the decoding strategy (greedy vs. sampling) for continuing the branch. If the prefix ends immediately before the flagged token, greedy decoding will regenerate the same token and hence the same original branch, making DDE equivalent to Vanilla with extra queries. Please provide the exact branch-generation procedure, including how the alternative token is chosen and how many alternatives are generated per branch point.","section":"§4.2, Figure 3"},{"comment":"The distance metric used in Representative Selection is undefined. Algorithm 1 computes 'avg. distance between s_i and all b in B' and 'avg. distance between s_i and all s_j in S', but no distance function is specified anywhere in the paper; BLEU, continuous token matching, and embedding similarity are defined in §5 only as evaluation metrics. Because the 'Closest' and 'Outlier' branches are the outputs of DDE, the selection rule is not reproducible without this definition.","section":"Algorithm 1"},{"comment":"The central claim that DDE 'consistently outperforms' Vanilla and DSR is not supported by any measure of statistical uncertainty. Table 6 reports a single value per setting, and the two ablation tables (Tables 9 and 10) are each based on 100 queries. Several improvements in Table 6 are small (e.g., I-R CodeLlama BLEU 'Full': Vanilla 0.651 vs. DDE 0.676), so confidence intervals or multiple random seeds are needed to establish that the average improvements of 9.96% and 5.73% are not noise.","section":"§6.1, Table 6"},{"comment":"The branch-point threshold τ is selected on the same evaluation setup used for the main results. Table 10 sweeps τ on the code domain and concludes that 0.8 is optimal, and the main results in Table 6 use τ = 0.8 on the same data. Because DDE's BLEU at τ = 0.2 (0.6127) is below the Vanilla baseline (0.651), the choice of τ is load-bearing; a held-out tuning procedure or a sensitivity analysis that does not select on the test set is needed.","section":"§6.4, Table 10"},{"comment":"The procedure for matching Vanilla's query count to DDE's branch count is underspecified. DDE generates multiple distinct branches per instruction, whereas greedy decoding is deterministic; if Vanilla repeats the same greedy output when given the same input, matching query counts does not give it additional chances to recover the ground truth. The paper should state whether Vanilla uses sampling (and at what temperature/decoding parameters) or whether the multiple queries are over different inputs, so that the fairness comparison is meaningful.","section":"§5, Compared Baselines"},{"comment":"The defense is claimed to 'fail DDE on the extracted data', but Figure 7 evaluates only the defended model's generation performance (HumanEval pass rates across temperature/top_p settings); there is no experiment measuring whether DDE's branch-point identification or extraction success is actually reduced. Without such an evaluation, the defensive contribution is not supported by the reported evidence.","section":"§7, Defense evaluation"}],"minor_comments":[{"comment":"The method name is inconsistent: 'Differentiate Data Extraction' is used in the Abstract and Section 1, while 'Differentiated Data Extraction' appears in the title and Section 4; please standardize.","section":"Title/Abstract"},{"comment":"There are several typographical issues: 'V anilla' appears with a space throughout (e.g., Abstract, §3.4), 'seperately' in Section 1 should be 'separately', and 'inlines' in Section 6.1 should be 'aligns' or 'is in line with'.","section":"Throughout"},{"comment":"The number of training epochs is not reported; the paper only states that the checkpoint with the lowest validation loss is used. Please add the epoch count or the early-stopping criterion.","section":"§5, Table 5"},{"comment":"The theoretical formula P(success) = 1 - (1 - NTC^k)^{(L-k+1)} is presented without defining k and L in the caption or surrounding text; please define these variables explicitly.","section":"Appendix F"},{"comment":"Each configuration in Tables 9 and 10 is based on 100 queries; please report the standard deviation or justify why this sample size is sufficient, especially because Table 10 is used to select τ.","section":"§6.4, Tables 9 and 10"},{"comment":"Since only 8 of 14 vendors currently return non-null logits and one vendor (Replicate) does not support the feature, the abstract's phrase 'all attack settings' should be qualified to settings where per-token logits are available; the current text in Section 3.1 is accurate but the abstract overstates the practical scope.","section":"Appendix I, Table 15"}],"recommendation":"major_revision","confidential_remarks":"No conflict of interest. The main concern is reproducibility: the branch-generation procedure and the distance metric in Algorithm 1 must be specified, and the defense needs an attack-success evaluation before publication. An artifact appendix with full pseudocode and raw per-condition results would substantially strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the first paper that actually defines the SFT-data-extraction problem, and the attack idea is worth engaging with. But the section that describes how DDE generates its branches is loose in a way that matters. If a careful reader can't tell whether the method forces alternative tokens at branch points or just re-queries the prefix, the reported 9.96% improvement over Vanilla isn't reproducible.\n\nWhat the paper does well: the formalization of attack goals (reconstruction vs. retraining), attack types (I-R vs. R-I), and partial-knowledge variants is a genuine step forward. The pilot NTC analysis is simple and useful. DDE's basic insight — use confidence gaps and the base model as a reference — is sensible. The experiments are broad: two domains, three preservation methods, three retention rates, two attack types, and an extra WildChat scalability test. On the numbers presented, DDE consistently beats Vanilla and DSR. The logits-vendor survey is a nice touch, and the defense, while limited, is at least described concretely.\n\nSoft spots. The stress-test about branch generation is fair. §4.1 says 'forcing M_FT with the second most probable token at potential deviation points,' but §4.2 doesn't say how the new branches are decoded or whether an alternative token is forced. Algorithm 1 references an 'average distance' but never defines it. That's a serious under-specification. Also: no error bars or significance tests anywhere; the main tables report single runs. τ=0.8 is chosen from a sweep on the same setup, so the tuned threshold could inflate gains. The defense evaluation only measures utility loss on HumanEval, not whether DDE actually fails against the modified logits, which is the claim that matters. Several results (Tables 9, 10) come from only 100 queries. And the token-match window of 25 is justified post hoc in Appendix F.\n\nNone of these are fatal to the core idea. The problem definition alone is interesting enough to warrant referee time. But the authors need to fix the branch-generation specification, add variance estimates, re-run the defense against DDE itself, and be transparent about τ and the match window.\n\nFor whom: security researchers working on LLM data leakage, and anyone building privacy defenses for fine-tuning APIs. It deserves a serious referee; I'd recommend conditional acceptance after major revision.","headline":"First solid formalization of targeted SFT extraction with a plausible attack, but the method's key branch-generation step is under-specified enough that the headline numbers should be treated with caution until the authors clarify it.","tokens_in":32671,"tokens_out":2587,"would_cite":true,"duration_ms":26171,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that supervised fine-tuning data—the valuable instruction-response pairs behind specialized LLM services—can be extracted from a deployed model through ordinary API queries, and that DDE, its differentiation-based method…","keywords":["supervised fine-tuning","data extraction attack","LLM privacy","token logits","branch deviation","reconstruction attack","retraining attack","instruction-response pairs"],"falsifier":"Apply DDE to a fine-tuned model served through an API that masks all token probabilities (the provider returns only the decoded string); if reconstruction BLEU falls to the Vanilla baseline's roughly 0.146 with 0% exact matches, the logits-based branch-point mechanism is the load-bearing component. Alternatively, compute NTC on a model fine-tuned on a private held-out dataset: if NTC is not clearly above the base model's next-token accuracy, the premise that SFT text remains retrievable is wrong.","tokens_in":31655,"feed_emoji":"🕵️","tokens_out":8910,"duration_ms":82248,"temperature":0.7,"pith_summary":"The paper's central claim is that the supervised fine-tuning (SFT) data used to specialize a large language model can be extracted from the deployed model by an attacker who only queries the API and reads per-token probabilities. Its proposed method, Differentiated Data Extraction (DDE), targets low-confidence token positions in the model's output, generates alternative continuations from those positions in both the fine-tuned model and an ordinary base model, and selects the continuation most likely to match the original training text. Across code and math domains, the paper reports that DDE beats straightforward querying by 9.96% and an earlier extraction method by 5.73% on reconstruction, and by 9.41% and 11.52% on retraining. If the claim holds, a fine-tuned model's most valuable asset—its proprietary instruction-response dataset—is recoverable through the model's own public interface.","feed_headline":"Fine-tuned LLMs leak private training pairs via token probabilities","feed_subtitle":"Reading token probabilities exposes the instruction-response pairs a fine-tuned model was trained on.","key_machinery":"The load-bearing mechanism is branch-point identification via a confidence threshold on per-token logits, followed by two-model branch comparison. DDE queries the fine-tuned model with greedy decoding, records the probability of each generated token, and flags any token below a threshold $\\tau$ as a potential branching point. It then continues generation from those prefixes in both the fine-tuned model and a base model, forming an SFT branch set and a base branch set; Algorithm 1 selects from the SFT set the branch closest on average to all base branches and the branch most distant from its own siblings. These two representatives, plus the masked-data completion step for retraining attacks, carry the entire argument that uncertain-generation behaviors expose the original training text.","core_discovery":"The central discovery is that fine-tuned models retain their SFT data at token level but ordinary generation rarely reproduces it because of branch deviation: one wrong token diverts the autoregressive process permanently. DDE removes that obstacle by using the fine-tuned model's confidence signal to find branch points, then comparing branches from the fine-tuned model with branches from a base model that lacks the SFT knowledge. From the fine-tuned model's branch set it picks two representatives, one closest to the base-model space and one most unlike its sibling branches, capturing underfitted and well-learned data respectively; a masked-data completion step then fills gaps in partially known instructions or responses. The paper reports an average Next Token Correction score of 0.8297, meaning about 83% of SFT tokens are recoverable if branch deviations are corrected, and presents experiments where DDE outperforms both baselines in every attack setting while matching the computational cost of simple querying.","pith_inferences":["If logits become unavailable, the natural escalation is empirical distribution estimation through many one-token queries; the paper names this adaptive route but does not quantify its cost, leaving an open measurement.","DDE's branch-deviation logic should transfer to parameter-efficient fine-tuning (for example, LoRA), since the method assumes only a fine-tuned model and a base model with different behavior, not full-parameter SFT.","The closest/outlier selection rule could double as a training-data auditing tool: low-confidence branch points mark tokens the model did not firmly memorize, while outlier branches mark tokens it did.","Because the attack needs only query access and logits, the value of SFT datasets themselves becomes an attack surface, shifting part of the privacy problem from model weights to API design."],"forward_implications":["A fine-tuned model served through an API can be forced to reproduce substantial portions of its proprietary SFT dataset, not just generic memorized text.","The extracted pairs are usable for retraining: models fine-tuned on DDE output approach the victim's benchmark performance, making SFT data a recoverable business asset.","Partial knowledge suffices: with a fraction of the instruction or response preserved, the missing side can be recovered, and higher retention rates generally improve recovery.","Attackers do not need the exact base model; different open models of similar capability retain most of DDE's advantage.","Providers can blunt DDE by rewriting returned logits to raise the top token's apparent confidence, leaving greedy decoding unchanged and most sampling behavior intact."],"supporting_citations":[{"why":"It supplies the untargeted Random attack baseline and the continuous-token matching metric that DDE adapts to a 25-token window.","marker":"[8]"},{"why":"It supplies the Poem attack baseline and the scalable pre-training data extraction context that DDE extends to targeted SFT extraction.","marker":"[9]"},{"why":"It provides DSR, the iterative beam-search extraction baseline that DDE must outperform in reconstruction attacks.","marker":"[4]"},{"why":"It provides LLaMA2-7B, the base and victim model for the math-domain and pilot experiments.","marker":"[19]"},{"why":"It provides CodeLlama-7B, the base and victim model for the code-domain experiments.","marker":"[32]"},{"why":"It supplies Alpaca-GPT4, the SFT dataset used in the pilot study that establishes branch deviation and NTC.","marker":"[26]"},{"why":"It supplies MagicCoder's OSS-Instruct, the code-domain SFT dataset and fine-tuning recipe.","marker":"[27]"},{"why":"It supplies MAmmoTH's MathInstruct, the math-domain SFT dataset.","marker":"[36]"},{"why":"It provides HumanEval, the benchmark used to measure retraining attack success in the code domain.","marker":"[42]"},{"why":"It provides GSM8K, the benchmark used to measure retraining attack success in the math domain.","marker":"[47]"}],"fun_headline_variants":["Token confidence reveals fine-tuned LLM training pairs","Confidence-based extraction recovers 83% of SFT tokens","New attack uses model confidence to steal fine-tuned data","Token probabilities leak SFT pairs from fine-tuned LLMs","DDE attack: confidence signals expose hidden training data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack collapses if the victim API does not return per-token logits or returns sanitized ones, because DDE needs those probabilities to identify branch points; the paper's vendor survey found that only 8 of 14 providers currently return non-null logits and one provider does not support the feature at all.","fun_headline_variants_meta":{"raw":{"variants":["Token confidence reveals fine-tuned LLM training pairs","Confidence-based extraction recovers 83% of SFT tokens","New attack uses model confidence to steal fine-tuned data","Token probabilities leak SFT pairs from fine-tuned LLMs","DDE attack: confidence signals expose hidden training data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000419,"raw_usage":{"total_tokens":2157,"prompt_tokens":944,"completion_tokens":1213,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":1131}},"tokens_in":560,"tokens_out":1213,"duration_ms":9012,"temperature":1.0,"reasoning_tokens":1131,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:20:33.779582+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply DDE to a fine-tuned model served through an API that masks all token probabilities (the provider returns only the decoded string); if reconstruction BLEU falls to the Vanilla baseline's roughly 0.146 with 0% exact matches, the logits-based branch-point mechanism is the load-bearing component. Alternatively, compute NTC on a model fine-tuned on a private held-out dataset: if NTC is not clearly above the base model's next-token accuracy, the premise that SFT text remains retrievable is wrong.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides GSM8K, the benchmark used to measure retraining attack success in the math domain."}],"review_version":2}