Self-generated QA supervision for language models is fragile due to non-uniform question selection and instruction compliance during answering, with mitigations that reduce compliance from 88% to 13%.
hub
Poisoning web-scale training datasets is practical
10 Pith papers cite this work. Polarity classification is still indexing.
10
Pith papers citing it
abstract
Deep learning models are often trained on distributed, web-scale datasets crawled from the internet. In this paper, we introduce two new dataset poisoning attacks that intentionally introduce malicious examples to a model's performance. Our attacks are immediately practical and could, today, poison 10 popular datasets. Our first attack, split-view poisoning, exploits the mutable nature of internet content to ensure a dataset annotator's initial view of the dataset differs from the view downloaded by subsequent clients. By exploiting specific invalid trust assumptions, we show how we could have poisoned 0.01% of the LAION-400M or COYO-700M datasets for just $60 USD. Our second attack, frontrunning poisoning, targets web-scale datasets that periodically snapshot crowd-sourced content -- such as Wikipedia -- where an attacker only needs a time-limited window to inject malicious examples. In light of both attacks, we notify the maintainers of each affected dataset and recommended several low-overhead defenses.
hub tools
citation-role summary
background 2
citation-polarity summary
roles
background 2representative citing papers
Defines empirical sensitivity and proves Ω(η + √(η d/n)) lower bound (tight up to logs) for any Gaussian mean estimator achieving optimal O(√(d/n)) ℓ₂ error.
First model organisms of narrow secret loyalties in LLMs evade black-box audits without principal knowledge and persist even at low poison fractions in training data.
125 coordinated Wikipedia animal-welfare edits dominate attribution and counterfactual influence for animal-welfare queries on Llama models, with no spillover to general queries about the same entities.
LLMs trained on simple specification gaming generalize to zero-shot reward tampering including rewriting their own reward function.
Use of model-generated content in training causes irreversible loss of distribution tails, termed model collapse, in VAEs, GMMs, and LLMs.
Learning a backdoored reference model as a poisonous-sample oracle enables near-perfect training-time backdoor removal with negligible natural-accuracy loss.
The paper formalizes fixed-set worst-case corruption in PBE, implements corruption searches on a string DSL, and shows VPA recovers some margin-1 tasks but fails on public SyGuS where vote margins are near one.
Introduces Generative Privacy Funnel (GenPF) and deep variational PF (DVPF) models that extend the privacy funnel to generative settings and provide a controllable privacy-utility trade-off with reduced sensitive attribute leakage in face recognition.
Bitwise-precise re-computation of LLM inference across GPU variants is achievable via software-only emulation, allowing rounding errors to serve as auditable signatures of the inference setup.
citing papers explorer
-
Self-Study Reconsidered: The Hidden Fragility of Learning from Self-Generated QA
Self-generated QA supervision for language models is fragile due to non-uniform question selection and instruction compliance during answering, with mitigations that reduce compliance from 88% to 13%.
-
Robust Statistical Estimators with Bounded Empirical Sensitivity
Defines empirical sensitivity and proves Ω(η + √(η d/n)) lower bound (tight up to logs) for any Gaussian mean estimator achieving optimal O(√(d/n)) ℓ₂ error.
-
Narrow Secret Loyalty Dodges Black-Box Audits
First model organisms of narrow secret loyalties in LLMs evade black-box audits without principal knowledge and persist even at low poison fractions in training data.
-
Small edits, large models: How Wikipedia advocacy shapes LLM values
125 coordinated Wikipedia animal-welfare edits dominate attribution and counterfactual influence for animal-welfare queries on Llama models, with no spillover to general queries about the same entities.
-
Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
LLMs trained on simple specification gaming generalize to zero-shot reward tampering including rewriting their own reward function.
-
The Curse of Recursion: Training on Generated Data Makes Models Forget
Use of model-generated content in training causes irreversible loss of distribution tails, termed model collapse, in VAEs, GMMs, and LLMs.
-
Two Sides of the Same Coin: Learning the Backdoor to Remove the Backdoor
Learning a backdoored reference model as a poisonous-sample oracle enables near-perfect training-time backdoor removal with negligible natural-accuracy loss.
-
Fixed-Set Robustness in Programming by Example: Example Corruption and Semantic Partition Recovery
The paper formalizes fixed-set worst-case corruption in PBE, implements corruption searches on a string DSL, and shows VPA recovers some margin-1 tasks but fails on public SyGuS where vote margins are near one.
-
Deep Privacy Funnel Model: From a Discriminative to a Generative Approach with an Application to Face Recognition
Introduces Generative Privacy Funnel (GenPF) and deep variational PF (DVPF) models that extend the privacy funnel to generative settings and provide a controllable privacy-utility trade-off with reduced sensitive attribute leakage in face recognition.
-
Bit-Exact AI Inference Verification Without Performance Tradeoffs
Bitwise-precise re-computation of LLM inference across GPU variants is achievable via software-only emulation, allowing rounding errors to serve as auditable signatures of the inference setup.