{"id":"10aea2f5-da3a-4d5a-acdd-c3b832606b4e","arxiv_id":"2412.09812","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ScaleOT uses reinforcement-learned layer importance, lightweight harmonizers, and selective rank compression to build privacy-preserving emulators for offsite tuning with near-lossless plug-in performance.","lead":"ScaleOT compresses large language models into smaller, weaker emulators that data owners can fine-tune without seeing the full model, then plugs the trained adapter back into the original model. The framework uses reinforcement-learned layer importance and low-rank compression to keep the final model close to full fine-tuning while making the emulator itself much less capable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Privacy claim rests solely on a performance gap under adapter-only tuning; the natural attack of full emulator fine-tuning is never evaluated, so 'better model privacy' is unsupported.","rationale":"I agree with the reader's weakest assumption: the paper equates privacy with a degraded emulator performance gap and never validates this against any actual adversary. My stress-test sharpens the concern by identifying the specific attack that would falsify the privacy claim: full fine-tuning of the emulator, which the protocol's 'Emulator FT' definition does not include. The paper's Section 3 conditions require Emulator FT < Plug-in, but that inequality is demonstrated only under adapter-only tuning; an adversary can choose a different fine-tuning procedure. The SRC method is particularly susceptible because low-rank initialization does not prevent an adversary from learning high-rank updates. The utility half of the central claim is credible: plug-in performance is close to or above full FT on medium models, and the large-model results are strong. Therefore the paper should not be rejected, but conditional acceptance with an adversarial fine-tuning experiment is appropriate. Since the reader's verdict is already CONDITIONAL and my concern reinforces that position, the verdict should remain unchanged.","tokens_in":15291,"tokens_out":4686,"duration_ms":51134,"concrete_test":"Run an adversarial fine-tuning experiment on GPT2-XL and OPT-1.3B using the exact emulators from Table 1. Give the adversary the emulator E* and the returned adapter A' as in the standard protocol, then allow full fine-tuning of all emulator parameters (including harmonizers and SVD-compressed layers) on the same downstream training data used by the data owner. Evaluate on the same eight benchmarks and compare against the reported Plug-in and full FT averages. Also run a second arm where the adversary initializes from A' plugged into E* and fine-tunes all parameters, testing whether the returned adapter provides a warm start. If the full-tuned emulator reaches within one average point of Plug-in or FT, the privacy claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central privacy claim is supported only by the observation that, under the protocol's prescribed adapter-only fine-tuning, the fine-tuned emulator underperforms the plug-in model (Table 1: GPT2-XL Emulator FT 45.0 vs Plug-in 49.3; OPT-1.3B 46.2 vs 49.9). The paper treats this gap as 'certify[ing] privacy protection' (Section 4.2), but the data owner is an adversary who is not obligated to tune only the adapter layers. A natural attack is to fine-tune the entire emulator, including the harmonizers and the SVD-compressed MHSA layers, on the downstream data, optionally starting from the returned adapter A'. The paper never evaluates this attack. The SVD compression in SRC is especially vulnerable: the low-rank matrices are merely initializations, and full fine-tuning can add full-rank updates that undo the rank restriction. If full emulator fine-tuning closes the gap to plug-in performance, the 'better model privacy' half of the central claim is false, even though the utility numbers may be correct. This is load-bearing because the paper's privacy argument (Section 3 conditions and Section 4.2) depends entirely on this gap persisting under adversarial fine-tuning, not just under the cooperative protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ScaleOT, an offsite-tuning framework for large language models that combines three components: a reinforcement-learning-based layer-importance estimator (Dynamic LayerReplace), lightweight 'harmonizer' networks that replace low-importance layers, and Selective Rank Compression (SRC), which applies SVD-based low-rank approximation to the MHSA weights of the retained layers. The framework is intended to generate emulators of varying sizes and compression levels so that data owners can fine-tune an adapter on the emulator and return it to the model owner, achieving plug-in performance close to full fine-tuning while keeping the emulator too weak to be a usable substitute. Experiments are reported on GPT-2-XL, OPT-1.3B/6.7B, and LLaMA-7B across WikiText and eight question-answering benchmarks, with ablations over the replacement ratio α, the rank compression ratio β, and the number of adapted layers Na.","tokens_in":15601,"tokens_out":5971,"duration_ms":65149,"significance":"If the utility results hold, ScaleOT is a useful step toward practical offsite tuning: it avoids the expensive knowledge-distillation phase of vanilla OT, provides a tunable privacy-utility knob through α and β, and is shown to be orthogonal to adapter-based PEFT methods. The empirical coverage across model families and tasks is a genuine strength, and the observation that MHSA rank compression degrades emulator performance faster than plug-in performance is interesting and potentially actionable. However, the paper's central privacy claim is not yet supported by evidence against an actual adversary. The authors operationalize privacy solely as a performance gap under the cooperative adapter-only tuning protocol and do not evaluate the natural attack of fine-tuning the full emulator, so the 'better model privacy' half of the abstract is currently an inference, not an established result. No code or model release is mentioned, which further limits reproducibility.","major_comments":[{"comment":"The privacy claim rests entirely on the observation that the fine-tuned emulator underperforms the plug-in model (e.g., Table 1: GPT2-XL Emulator FT 45.0 vs Plug-in 49.3; OPT-1.3B 46.2 vs 49.9). This observation is obtained under the cooperative protocol that fine-tunes only the adapter layers. An adversary who obtains the emulator is not obliged to follow this protocol: she can fine-tune the entire emulator, including the harmonizers and the SVD-compressed MHSA layers, optionally initializing from the returned adapter. The SVD factors are merely initializations, and full-rank fine-tuning can undo the rank restriction. The statement in Section 4.2 that SRC 'certifies privacy protection' is therefore unsupported. Please define a threat model and evaluate at least the full-emulator fine-tuning attack (and ideally a distillation or model-extraction baseline) for GPT2-XL and OPT-1.3B; if the gap closes under this attack, the 'better model privacy' claim should be revised or qualified.","section":"§3, §4.2, Table 1"},{"comment":"No variance, confidence intervals, or repeated runs are reported anywhere in the empirical evaluation. The differences that support 'nearly lossless' and the α/β trends are often only 1–3 points on task averages (e.g., Table 1: ScaleOT Plug-in 49.3 vs Full FT 49.9 for GPT2-XL), which is within the typical run-to-run noise of fine-tuning experiments. Please report mean ± std over at least three random seeds for the main tables and for the Fig. 4 trade-off curves, or otherwise justify that seed variance is negligible for these settings.","section":"§5, Tables 1–3, Figs. 3–4"},{"comment":"The importance-score update in Eq. (5) is a heuristic reward-weighted gradient step rather than a standard policy-gradient or otherwise principled RL update, and no objective, convergence guarantee, or analysis is provided. Because the entire Dynamic LayerReplace procedure and the subsequent emulator creation depend on these scores, the paper should provide evidence that the scores are reliable and stable: for example, report the RL reward/loss curve over training, an ablation over the number of candidate networks Nc, and a stability check such as the correlation between importance scores from independent training runs or agreement with an oracle importance measure like layer-wise gradient norms.","section":"§4.1, Eq. (5)"},{"comment":"The 'nearly lossless offsite tuning compared with full fine-tuning' claim cannot be verified for the large models: Table 2 reports zero-shot and plug-in results for OPT-6.7B and LLaMA-7B but no full fine-tuning numbers. In addition, the CRaSh baseline is evaluated only on the large models, so the medium-model comparison in Table 1 omits a strong recent baseline. Please add the missing full-FT baselines for the large models and CRaSh results on GPT2-XL and OPT-1.3B, or explicitly state that those runs were not performed and qualify the lossless claim accordingly.","section":"§5, Table 2"}],"minor_comments":[{"comment":"The word 'definations' should be 'definitions'.","section":"Preliminary"},{"comment":"The caption 'with varies β' should read 'with various β'.","section":"Fig. 3 caption"},{"comment":"The symbol π_i is used both for a distribution and for the sampled probability p_i; please use distinct notation, for example π_i for the policy and p_i for the realized sample.","section":"Eq. (1)"},{"comment":"The relationship between k = Na/Ng and κ = n×α/Ng is not explained, and it is not immediately obvious that the harmonizer index set φ_H is always a subset of φ_E for arbitrary α and Na; please clarify the intended inequalities and ranges.","section":"Eqs. (8)–(10)"},{"comment":"The reference 'et al., R. P. 2024' is malformed; it should be 'Pan, R., et al. 2024' (the LISA paper).","section":"References"},{"comment":"In Table 3 the column headers 'Emu. FT ↑' and 'Plug-in ↓' are confusing because lower perplexity is better; the arrows seem to denote desired directions for different columns, so please clarify.","section":"Table 3"},{"comment":"The phrase 'certify privacy protection' is too strong for a performance-gap observation; consider replacing it with a more operational statement such as 'increases the gap under the protocol described.'","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The utility contribution is plausible and the empirical scope is decent, but the privacy claim needs a real adversary evaluation before publication. The most important missing experiment is full-emulator fine-tuning (including the harmonizers and SVD-compressed layers) with and without the returned adapter as initialization. If that attack closes the gap, the abstract's 'better model privacy' claim should be significantly weakened. The paper should also report variance over seeds and add the missing full-FT/CRaSh baselines. These are addressable within the manuscript's scope, so I recommend major revision rather than rejection. The topic is well within the scope of a CS/CL venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ScaleOT is a useful engineering contribution that makes offsite tuning cheaper and more flexible, but the privacy half of the central claim is not actually tested. What's new is the combination of RL-estimated layer importance, trainable harmonizers, and selective SVD rank compression on MHSA layers. That combination is not in OT, CRaSh, or LISA, and the experimental sweep across GPT-2-XL, OPT-1.3B/6.7B, and LLaMA-7B is substantially broader than what OT reported. The plug-in numbers are genuinely good: near full fine-tuning, sometimes above it, across eight tasks. That part I'd trust.\n\nThe soft spot is the privacy claim. Section 4.2 says SRC 'certifies privacy protection,' but the only evidence is that Emulator FT < Plug-in under the prescribed adapter-only tuning protocol. That does not certify anything. A data owner who wants the model is not bound to tune only the adapter; they can fine-tune the whole emulator, including the harmonizers and the low-rank SVD layers. The SVD matrices are initializations; full fine-tuning can add full-rank updates. The paper never evaluates this trivial attack, nor any distillation or extraction attack. If full emulator fine-tuning closes the gap, the 'better model privacy' claim is false. This is load-bearing, so the paper overstates what it has shown.\n\nSmaller issues: no variance or seeds, no code release, and the default alpha and beta are chosen by inspecting the same curves used as evidence (Figure 4). That is a modicum of selection, not fatal, but worth noting. The RL heuristic is fine as an empirical choice; the lack of an attack model is the real gap.\n\nThis paper is for people building practical offsite-tuning pipelines, not for security researchers who care about rigorous threat models. The central utility result is solid and worth building on. I would send it to a serious referee—it deserves referee time—but I would expect major revision. If you work on offsite tuning, cite it for the method and the plug-in numbers; do not cite it for privacy protection until the threat model is actually exercised.","headline":"Useful offsite-tuning method with strong plug-in numbers, but the privacy claim is an assertion, not a result—full emulator fine-tuning is never tested.","tokens_in":16113,"tokens_out":2854,"would_cite":true,"duration_ms":29546,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ScaleOT claims to deliver nearly lossless offsite tuning—plugging the returned adapter into the original model matches full fine-tuning—while the shared emulator is degraded enough to deter theft.","keywords":["offsite-tuning","large language models","privacy-preserving fine-tuning","layer importance","reinforcement learning","low-rank compression","harmonizers","emulator"],"falsifier":"Give an adversary the emulator, its harmonizers, and the returned adapter, allow distillation on a public proxy dataset, and check whether the recovered model's accuracy approaches full fine-tuning; if it does while the emulator's own zero-shot and fine-tuned scores remain low, the privacy claim collapses.","tokens_in":15114,"feed_emoji":"🔐","tokens_out":6810,"duration_ms":64998,"temperature":0.7,"pith_summary":"This paper tries to establish a compression scheme for offsite-tuning that fixes two weaknesses of the standard approach: uniform layer dropping degrades the final adapted model, and knowledge distillation is too expensive. ScaleOT instead learns which transformer layers matter via reinforcement learning, replaces unimportant layers with lightweight harmonizers, and applies SVD low-rank compression only to attention matrices. The authors report that the resulting emulator can be fine-tuned by a data owner and the returned adapter, plugged back into the full model, performs about as well as full fine-tuning—sometimes slightly better—while the emulator itself scores much worse, which they take as model privacy. The practical payoff would be a cheap, adjustable way to let third parties fine-tune large models without exposing the weights or the data.","feed_headline":"ScaleOT tunes LLMs offsite with nearly lossless accuracy","feed_subtitle":"Adapting the original model's adapter yields full-fine-tune accuracy while the shared emulator stays too weak to steal.","key_machinery":"The load-bearing mechanism is Dynamic LayerReplace combined with Selective Rank Compression. Dynamic LayerReplace learns a per-layer importance score $s_i$ via reinforcement learning: candidate networks are formed by replacing half the layers in each of $N_g = 4$ groups with harmonizers (lightweight low-rank FFNs), rewards are $r_j = e^{-L_j} - \\frac{1}{N_c}\\sum_t e^{-L_t}$, and scores update by $s_i \\leftarrow s_i + r_j \\sigma(s_i)(1-\\sigma(s_i))$ for layers in the sampled set. The emulator is then composed of harmonizers for the least important layers and, for the remaining frozen layers, SVD-compressed attention matrices with rank reduction ratio $\\beta$; the adapter is the top $N_a/N_g$ layers per group. This machinery creates a deliberate asymmetry: attention rank compression degrades the emulator's own fine-tuned accuracy quickly while the plug-in performance declines slowly, which is what turns compression into a privacy lever rather than just a utility cost.","core_discovery":"The central claim is that a lossy emulator built by replacing low-importance transformer layers with lightweight harmonizers chosen by a reinforcement-learning importance score, and then applying SVD low-rank compression only to multi-head attention matrices, can be fine-tuned by a data owner whose returned adapter restores the original model to nearly full-fine-tuning accuracy. On multiple benchmarks the plug-in performance matches or slightly exceeds direct fine-tuning, while the emulator's own zero-shot and fine-tuned scores fall substantially. The authors read this performance gap as model privacy: a downstream user who keeps the emulator gets a weaker model, and is therefore encouraged to return the adapter and use the plugged-in original.","pith_inferences":["The paper's privacy claim rests on the emulator's weak task scores; it does not run extraction, distillation, or inversion attacks, so the actual privacy guarantee under an adaptive adversary is undetermined.","The finding that tuning only important layers can beat full fine-tuning suggests the same reinforcement-learned importance scores could be repurposed for memory-efficient fine-tuning of larger models, a connection the authors note but do not develop into a method.","Selective Rank Compression's choice to compress attention but not feedforward weights is an empirical asymmetry; testing whether the same asymmetry transfers to other architectures, such as vision or multimodal transformers, would be a natural check.","The returned adapter itself may carry information about the original layers; analyzing whether the adapter can be combined with the emulator to reconstruct a higher-performing model would tighten the privacy argument."],"forward_implications":["Plug-in accuracy roughly matches full fine-tuning on the tested QA and language-modeling tasks, and on several benchmarks it slightly exceeds it, so downstream users lose little by using the returned adapter.","Because emulator zero-shot and fine-tuned performance drop while plug-in stays high, the shared artifact satisfies the three conditions the paper defines for effective offsite tuning: zero-shot below plug-in, emulator fine-tuning below plug-in, and plug-in close to full fine-tuning.","The two knobs $\\alpha$ (harmonizer replacement ratio) and $\\beta$ (rank reduction ratio) give a privacy-utility frontier, letting a model owner issue emulators of different scales rather than one fixed compressed model.","ScaleOT composes with parameter-efficient methods: adding LoRA (rank 4) or adapters (dimension 64) cuts trainable parameters to below 2 million while preserving plug-in perplexity.","Compression requires training only about 1–2% of the model's parameters, which the paper argues makes offsite tuning practical for models in the 7-billion-parameter range without knowledge distillation."],"supporting_citations":[{"why":"Defines offsite-tuning and the Uniform LayerDrop baseline that ScaleOT compares against, along with the three performance conditions the paper adopts.","marker":"Xiao, Lin, and Han 2023"},{"why":"Supplies the LayerDrop technique that vanilla OT uses; ScaleOT replaces uniform dropping with importance-aware replacement.","marker":"Sajjad et al. 2023"},{"why":"Provides the SVD rank-$r$ optimality theorem underpinning Selective Rank Compression.","marker":"Eckart and Young 1936"},{"why":"Supplies The Pile corpus used to train Dynamic LayerReplace importance scores and harmonizers.","marker":"Gao et al. 2020"},{"why":"Provides the OPT model family used in the medium and large model evaluations.","marker":"Zhang et al. 2023b"},{"why":"Provides LLaMA-7B, one of the large-scale testbeds for ScaleOT.","marker":"Touvron et al. 2023"},{"why":"LoRA is the parameter-efficient method combined with ScaleOT in the orthogonality experiments.","marker":"Hu et al. 2022"},{"why":"Supports the claim that layer importance varies and that tuning a subset of layers can beat full fine-tuning.","marker":"Yao et al. 2024"},{"why":"Provides empirical evidence behind sparse training convergence that ScaleOT invokes to explain plug-in performance exceeding full fine-tuning.","marker":"Pan et al. 2024"}],"fun_headline_variants":["RL-picked layer swaps make private tuning nearly lossless","ScaleOT: private tuning with full-fine-tune accuracy, no distillation","Weak emulator, strong adapter: offsite tuning that scales privacy","ScaleOT: selective compression for private offsite tuning with near-zero loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That a noticeably weaker emulator, measured by benchmark accuracy, cannot be used by an adversary to extract or reconstruct a usable model from the returned adapter.","fun_headline_variants_meta":{"raw":{"variants":["RL-picked layer swaps make private tuning nearly lossless","ScaleOT: private tuning with full-fine-tune accuracy, no distillation","Weak emulator, strong adapter: offsite tuning that scales privacy","ScaleOT: selective compression for private offsite tuning with near-zero loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1508,"prompt_tokens":900,"completion_tokens":608,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":532}},"tokens_in":516,"tokens_out":608,"duration_ms":6561,"temperature":1.0,"reasoning_tokens":532,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:41:25.471734+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give an adversary the emulator, its harmonizers, and the returned adapter, allow distillation on a public proxy dataset, and check whether the recovered model's accuracy approaches full fine-tuning; if it does while the emulator's own zero-shot and fine-tuned scores remain low, the privacy claim collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LayerDrop technique that vanilla OT uses; ScaleOT replaces uniform dropping with importance-aware replacement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the claim that layer importance varies and that tuning a subset of layers can beat full fine-tuning."}],"review_version":1}