REVIEW 19 cited by
Stealing Part of a Production Language Model
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Stealing Part of a Production Language Model
read the original abstract
We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI's ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model, given typical API access. For under \$20 USD, our attack extracts the entire projection matrix of OpenAI's Ada and Babbage language models. We thereby confirm, for the first time, that these black-box models have a hidden dimension of 1024 and 2048, respectively. We also recover the exact hidden dimension size of the gpt-3.5-turbo model, and estimate it would cost under $2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possible future work that could extend our attack.
Forward citations
Cited by 19 Pith papers
-
Channel Location Constrains the Auditability of Subliminal Learning
Auditability of subliminal learning is constrained by channel location, with initialization-dependent body channels allowing pre-training screens while vocabulary geometry and conditional body channels evade them.
-
SoK: Colluding Adversaries in Machine Learning Pipelines
The paper introduces a framework for collusion between train- and inference-time adversaries in ML pipelines, proposes a guideline for conjecturing collusion potential, explains prior work, and empirically validates f...
-
Unlearning with Asymmetric Sources: Improved Unlearning-Utility Trade-off with Public Data
ALU uses public data to suppress unlearning cost quadratically while characterizing distribution mismatch effects, enabling mass unlearning with maintained utility.
-
Unlearning with Asymmetric Sources: Improved Unlearning-Utility Trade-off with Public Data
Asymmetric Langevin Unlearning uses public data to suppress unlearning noise costs by O(1/n_pub²), enabling practical mass unlearning with preserved utility under distribution mismatch.
-
A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Framework
A new 7x4 taxonomy organizes agentic AI security threats by architectural layer and persistence timescale, revealing under-explored upper layers and missing defenses after surveying 116 papers.
-
Fingerprinting LLMs via Prompt Injection
LLMPrint generates unique, post-processing-robust fingerprints for base LLMs and their variants via optimized prompt injection with statistical verification for gray-box and black-box settings.
-
Can Watermarking Techniques Help Prevent LLM Model Stealing?
Softplus-then-perturb with embedding-seeded Gaussian noise defeats PCA/averaging/RPCA dimension-extraction attacks on Mistral-7B and GPT-2 with only modest quality loss.
-
Tracing LLM Behavior to the Training Data with Empirical Next-Token Distributions
Across Pythia models, a large fraction of training contexts are predicted almost exactly by the empirical next-token distribution of the corpus, but frequent high-entropy contexts remain poorly matched.
-
Tracing LLM Behavior to the Training Data with Empirical Next-Token Distributions
A trained transformer's next-token distribution often matches the empirical next-token distribution of its pretraining corpus, with agreement improving as models grow, while a persistent tail of mismatches remains.
-
Black-Box Inference of LLM Architectural Properties with Restrictive API Access
NightVision recovers LLM hidden dimension to 23% average relative error (9% on MoE) and depth/parameter count to 53% on models >3B parameters using common-set prompting, spectral analysis, and TTFT under single-logit ...
-
Surrogate Fidelity: When Can Open LLMs Explain Closed Ones?
Prediction agreement between open and closed LLMs substantially overstates agreement on attributions and causal reasons.
-
OTRO: Oblivious Tokenization Path with Square-Root ORAM
OTRO combines replicated square-root ORAM instances, epoch rotation with dummy padding, and KV-cache-aware chunking to make tokenizer lookups oblivious with at most 4.5% TTFT overhead and under 0.5 GB extra memory in ...
-
The Surface You Test Is Not the Surface That Breaks
Prompt injection vulnerability in tool-augmented LLMs is a model-surface interaction rather than a fixed channel property; the same payload inverts success rates across models, and adaptive attack rate exceeds single-...
-
On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference
An attack aligns differently shuffled intermediate activations from secure Transformer inference queries to recover model weights with low error using roughly one dollar of queries.
-
Characterizing Linear Alignment Across Language Models
Linear (affine) maps between final hidden states of independent LLMs preserve downstream performance and can enable text generation when tokenizers and scale align, enabling a practical HE-based privacy protocol.
-
CrypTorch: PyTorch-based Auto-tuning Compiler for Machine Learning with Multi-party Computation
An MPC-ML compiler that modularizes and auto-tunes operator approximations, delivering 1.2–1.8x speedups over an optimized baseline under user-set accuracy bounds.
-
LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Users
A single attacker can use strategic upvoting and downvoting on language model outputs to inject facts, security flaws, or fake news that persist in the model for all users after preference tuning.
-
How Well Do AI Systems Solve AP Physics? A Comparative Evaluation of Large Language Models on Algebra-Based Free Response Questions
ChatGPT 4.1 mini, Gemini 2.5 Flash, Claude 4.0 Sonnet, and DeepSeek R1 average 82–92% on AP Physics 1/2 free-response questions but systematically fail spatial, visual, and conceptual tasks.
-
The Geometry of Last-Layer Model Stealing
Geometry maps the conditions for perfect last-layer theft in transformers and demonstrates that full hidden-network reverse engineering is impossible from final outputs.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.