REVIEW 15 cited by
Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The problem of pre-training data detection for large language models (LLMs) has received growing attention due to its implications in critical issues like copyright violation and test data contamination. Despite improved performance, existing methods (including the state-of-the-art, Min-K%) are mostly developed upon simple heuristics and lack solid, reasonable foundations. In this work, we propose a novel and theoretically motivated methodology for pre-training data detection, named Min-K%++. Specifically, we present a key insight that training samples tend to be local maxima of the modeled distribution along each input dimension through maximum likelihood training, which in turn allow us to insightfully translate the problem into identification of local maxima. Then, we design our method accordingly that works under the discrete distribution modeled by LLMs, whose core idea is to determine whether the input forms a mode or has relatively high probability under the conditional categorical distribution. Empirically, the proposed method achieves new SOTA performance across multiple settings. On the WikiMIA benchmark, Min-K%++ outperforms the runner-up by 6.2% to 10.5% in detection AUROC averaged over five models. On the more challenging MIMIR benchmark, it consistently improves upon reference-free methods while performing on par with reference-based method that requires an extra reference model.
Forward citations
Cited by 15 Pith papers
-
Learning the Signature of Memorization in Autoregressive Language Models
A classifier trained only on transformer fine-tuning data detects an invariant memorization signature that transfers to Mamba, RWKV-4, and RecurrentGemma with AUCs of 0.963, 0.972, and 0.936.
-
Crossing the Margin Cliff: Toward Relearn-Robust LLM Unlearning via Margin Calibration
Margin Calibration, a non-saturating margin-anchored LoRA polish, crosses the margin cliff and cuts post-attack relearn recovery on all 97 populated cells in the paper's stress matrix.
-
Averaged Evaluation Masks Capability Trade-Offs: Multi-Source Calibration for High-Sparsity LLM Pruning
Analysis of 15 calibration sources shows opposite-sign Spearman correlations between perplexity and retention across General vs. Math/Code dimensions in LLM pruning, and multi-source mixing via IGSP raises total reten...
-
Implicit Reasoning Steering via Concept Chaining
Reinforcement-learning-optimized concept-chain paragraphs covertly steer language-model multiple-choice preferences after continued pretraining, with far lower detectability than direct paraphrases.
-
Amplifying Membership Signal Through Chained Regeneration
MADreMIA amplifies membership inference signals by showing that memorized samples maintain higher coherence and slower degradation in chained regeneration trajectories than non-members.
-
MC-PDD: Masked Corpus-Level Pretraining Data Detection for Black-Box Large Language Models
A masked-token hit-rate comparison method detects pretraining data membership in black-box LLMs with performance comparable to white-box approaches.
-
Averaged Evaluation Masks Capability Trade-Offs: Multi-Source Calibration for High-Sparsity LLM Pruning
At 60% SparseGPT sparsity, mixing four capability-matched calibration corpora preserves 58.8% of unpruned performance versus 40.0% for C4 — but the mix is partially calibrated on the same GSM8K and MBPP benchmarks use...
-
TRACER: A Semantic-Aware Framework for Fine-Grained Contamination Detection in Code LLMs
TRACER presents a semantic-aware framework and the first benchmark for fine-grained code contamination detection across three levels of overlap, reporting F1 scores of 0.91-0.92 and large gains over prior methods.
-
Lost in Modality: Evaluating the Effectiveness of Text-Based Membership Inference Attacks on Large Multimodal Models
Logit-based MIAs perform comparably on MLLMs with or without visual inputs in-distribution but visual inputs mask membership signals in out-of-distribution settings.
-
Auditing Data Membership in Reinforcement Learning With Verifiable Rewards
DIBA detects membership of prompts in RLVR training by measuring reward success changes and policy behavioral drift between pre- and post-RLVR model checkpoints.
-
LLM generation novelty through the lens of semantic similarity
An output is called novel when no semantically similar pretraining context can be retrieved; using this embedding-based measure, SmolLM models show longer-range reuse and instruction tuning increases novelty.
-
Investigating Training Data Detection in AI Coders
Most existing training-data detection methods perform poorly on code, while prefix-relative method ReCaLL consistently scores highest, though all degrade under code mutations.
-
Identifying Pre-training Data in LLMs: A Neuron Activation-Based Detection Framework
NA-PDD detects pre-training data in LLMs by comparing which neurons activate for a test text against neurons linked to known training versus non-training texts, and claims large AUC improvements on three benchmarks.
-
Membership Inference Attacks on Tokenizers of Large Language Models
Tokenizers leak dataset membership: using distinctive tokens that appear in a dataset's text, an attacker can detect with AUC up to 0.77 whether that dataset was part of tokenizer training.
-
Neural Breadcrumbs: Membership Inference Attacks on LLMs Through Hidden State and Attention Pattern Analysis
A random forest trained on transformer hidden-state and attention features detects training data membership with about 0.83 average AUC, far above output-based attacks.
Discussion (0). Sign in to comment.