REVIEW 6 cited by
Offsite-Tuning: Transfer Learning without Full Model
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Transfer learning is important for foundation models to adapt to downstream tasks. However, many foundation models are proprietary, so users must share their data with model owners to fine-tune the models, which is costly and raise privacy concerns. Moreover, fine-tuning large foundation models is computation-intensive and impractical for most downstream users. In this paper, we propose Offsite-Tuning, a privacy-preserving and efficient transfer learning framework that can adapt billion-parameter foundation models to downstream data without access to the full model. In offsite-tuning, the model owner sends a light-weight adapter and a lossy compressed emulator to the data owner, who then fine-tunes the adapter on the downstream data with the emulator's assistance. The fine-tuned adapter is then returned to the model owner, who plugs it into the full model to create an adapted foundation model. Offsite-tuning preserves both parties' privacy and is computationally more efficient than the existing fine-tuning methods that require access to the full model weights. We demonstrate the effectiveness of offsite-tuning on various large language and vision foundation models. Offsite-tuning can achieve comparable accuracy as full model fine-tuning while being privacy-preserving and efficient, achieving 6.5x speedup and 5.6x memory reduction. Code is available at https://github.com/mit-han-lab/offsite-tuning.
Forward citations
Cited by 6 Pith papers
-
How Context Attribution Handles What the Model Already Knows
Context attribution methods cannot disentangle in-context from in-weight knowledge and assign unfaithful scores under overlap; new metrics and WMDP-Cyber++ quantify the failure.
-
Harnessing LLMs for Document-Guided Fuzzing of OpenCV Library
VISTAFUZZ, an LLM-based document-guided fuzzer, tested 330 OpenCV-python APIs and produced 17 bug reports, with 10 confirmed and 5 fixed by maintainers.
-
GradOT: Training-free Gradient-preserving Offsite-tuning for Large Language Models
GradOT selects compression by minimizing a Gradient-preserving Compression Score derived from a Taylor approximation of the offsite-tuning objective, achieving competitive plug-in performance and larger emulator-to-pl...
-
Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
POST tunes a soft prompt privately on a small distilled model and transfers it to the large model using a two-term public-data loss, with optional differential privacy.
-
Adapt Once, Thrive with Updates: Transferable Parameter-Efficient Fine-Tuning on Evolving Base Models
Trans-PEFT uses random FFN masking and layer dropping during fine-tuning so PEFT modules trained on an old base model transfer to a continually updated base model without retuning.
-
ISACL: Internal State Analyzer for Copyrighted Training Data Leakage
An MLP trained on LLM internal states predicts Rouge-L-defined literal copying leakage with high accuracy, but not paraphrase-level leakage.
Discussion (0). Sign in to comment.