Pith. sign in

REVIEW 3 major objections 6 minor 40 references

NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptation

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that NoEsis, a hybrid of DP-trained prompt tokens and non-private Mix-LoRA experts, can deliver document-level differential privacy, knowledge transfer, and modularity at the same time, backed by code-completion…

desk verdict Empirically solid hybrid of DP prompt tuning and Mix-LoRA, but the privacy claim covers only the shared prompt tokens; the per-client guarantee is plausible, unstated, and needs to be proved. read the letter →

arxiv 2504.18147 v1 pith:YHW5W7B4 submitted 2025-04-25 cs.CR cs.LG

Rob Romijnders , Stefanos Laskaridis , Ali Shahin Shamsabadi , Hamed Haddadi This is my paper · ORCID
classification cs.CRcs.LG
keywords DifferentialprivacyKnowledgetransferModularLLMsLoRAexpertsPrompttuningMembershipinferenceattacksCodecompletionMulti-domainadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that modular LLMs can have privacy, modularity, and knowledge transfer at once, defusing a trade-off that earlier approaches left unresolved. It proposes NoEsis, where a small number of trainable prompt tokens are trained with differential privacy and shared across domains, while each domain gets its own non-private LoRA expert. On multilingual code completion, the method keeps document-level privacy at epsilon equal to 1 and delta equal to 1e-6 and still bridges at least 77% of the accuracy gap between having no sharing and having no privacy. If true, this means organizations with separate private codebases can train one modular model and serve per-domain copies without leaking each other's training data.

What carries the argument

The machinery is a two-stage training loop built from two parameter groups. Stage 1 trains a small matrix of prompt embeddings P, prepended to every domain's input, using DP-SGD with per-document gradient clipping and Gaussian noise, so P satisfies document-level (epsilon, delta)-DP. Stage 2 keeps P fixed and trains a Mix-LoRA per domain: low-rank matrices A(k) and B(k) update the frozen backbone through W = W0 + alpha times the sum of B(k)A(k), applied to the feed-forward layers. Deployment merges each expert into the backbone and distributes the shared prompt tokens to every client. The separation between shared-but-private parameters and private-but-non-shared experts is what carries the argument for both utility and privacy.

What would settle it

Train NoEsis twice on the same data with the same hyperparameters and random seed, once with the full training set and once with a single document removed from one domain, then measure how much the final shared prompt-token matrix changes. If the change exceeds what the chosen epsilon equal to 1 and delta equal to 1e-6 accountant permits for a single document, then the training procedure is not actually implementing the claimed document-level differential privacy.

Watch

Extended reading notes

Core claim

The central claim is that the apparent conflict between differential privacy and multi-domain knowledge transfer can be resolved by splitting parameters by role: shared knowledge lives in a small set of DP-trained prompt tokens, while domain-specific skills live in non-private LoRA adapters that are never shared. The paper demonstrates this split on CodeXGLUE code completion: 32 shared prompt tokens outperform a shared LoRA with comparable parameter counts, the domain experts supply the bulk of accuracy, and the DP guarantee on the prompts empirically pushes cross-domain membership inference toward random chance. The paper presents NoEsis as the first framework satisfying modularity, privacy, and knowledge transfer simultaneously, with the strongest quantitative claim being that it closes at least 77% of the accuracy gap between a non-shared baseline and a non-private baseline on code completion.

Load-bearing premise

The whole-system privacy claim rests on the assumption that the domain-specific LoRA experts, trained without any privacy protection and never shared, cannot be combined with the shared prompt tokens to leak information beyond the differential privacy guarantee on those tokens.

Editorial extensions

If this is right

  • If NoEsis is right, modular LLMs can be fine-tuned per domain without giving up knowledge transfer, because the shared knowledge can live in a tiny DP-trained prompt set while each domain's adaptation stays in its own private expert.
  • On code completion, the shared prompt tokens are a parameter-efficient transfer mechanism: 32 tokens outperform a shared LoRA at four times the parameter count, and larger prompt sets do not help further.
  • Scarce domains benefit the most from transfer: Go, which has roughly 50 times fewer training documents than Python, gains 2.41 accuracy points over a share-nothing modular model.
  • Standard Mix-LoRA leaks across domains, with a cross-domain membership inference attack reaching AUC 65.4% when attacking Java from Go, while NoEsis reduces that attack to 55.7%, closer to random guessing.
  • Deployment is modular: each domain's LoRA is merged into the shared backbone and the same DP-trained prompt tokens are sent to all clients, so training once can serve every domain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference beyond the paper: the two-stage split suggests a general recipe, namely train any small shared interface (prompts, prefixes, or a low-rank common adapter) with DP and keep domain experts non-private; the paper tests prompts, but the same logic could apply to other shared parameter sets if the privacy accountant is run on them.
  • Inference beyond the paper: the reported membership inference protection is for likelihood-threshold attacks; the paper's own limitations section names adversarial and reconstruction attacks as open, so empirical privacy should be read narrowly until those attack surfaces are also tested.
  • Inference beyond the paper: because the DP guarantee is not proven end-to-end over the non-private experts, a stronger privacy claim would require either private expert gradients or an explicit composition argument; the experiments support utility plus empirical MIA protection more strongly than a formal guarantee on the whole served model.
  • A testable extension: apply the same hybrid to natural-language domains or to alternative adapters such as DoRA or VeRA; if the 77% gap-bridging pattern persists, the parameter-splitting mechanism is general, not a quirk of code completion.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces NoEsis, a two-stage parameter-efficient fine-tuning framework for multi-domain LLM adaptation that combines DP-trained shared prompt tokens (Stage 1) with non-private per-domain LoRA experts (Stage 2). On CodeXGLUE code completion across Python, Java, and Go, it reports higher accuracy than share-nothing and prompt-tuning-only baselines, closes at least 77% of the accuracy gap to the non-private baseline, and lowers cross-domain membership-inference attack AUC relative to a Mix-LoRA baseline. The authors claim document-level DP at ε=1, δ=1e-6 and position NoEsis as the only approach that is simultaneously modular, private, and able to transfer knowledge across domains.

Significance. If the intended privacy guarantee is stated precisely and proved, this is a useful and timely contribution: it offers a concrete architecture for serving per-domain adapters with a shared DP-trained prompt backbone, and it ships with reproducible code, detailed hyperparameter reporting, multi-seed variability analysis, and ablations. The empirical utility results are internally consistent with the plotted numbers, and the code-release and experimental documentation are strengths. The main gap is formal: as written, the DP claim is broader than the released artifacts actually satisfy, and the missing theorem is load-bearing because the paper's claimed uniqueness rests on the privacy property. The central claim is defensible and likely fixable by restating the guarantee as a per-client cross-domain guarantee and proving it.

major comments (3)
  1. [Section 3, Algorithm 1, Table 1, Abstract] The paper's privacy claim is not supported for the artifacts that are actually released. Under the adjacency definition in Section 3, two datasets are adjacent when one document in any domain is removed. Stage 1 makes the prompt tokens P_T differentially private with respect to the full dataset, but Stage 2 trains each domain expert E_k non-privately on its own domain. For an adjacent pair differing in a document of domain c, the released expert E_c changes arbitrarily, so the full tuple (P_T, E_1, ..., E_K) is not (1, 1e-6)-DP. The statement that is both true and likely intended is a per-client cross-domain guarantee: a client in domain c receives (P_T, E_c), and for adjacent datasets differing only outside domain c, E_c is a post-processing function of P_T with D_c fixed, so the pair is (ε,δ)-DP. This theorem is neither stated nor proved, and the unqualified phrases 'provable privacy guarantees' and Table 1's 'Private ✓' therefore overstate what is established.
  2. [Algorithm 1, Stage 2] The pseudocode for Stage 2 is ambiguous in a way that matters for privacy. It says to 'Randomly draw batch B_t of size N_b from D' and update the full expert parameter set E_t with SGD. Read literally, this would update every domain expert on batches that may contain documents from other domains, which would destroy even the per-client cross-domain guarantee analyzed above. The surrounding text says experts are per-domain, so the intended behavior is presumably that only the expert matching each document's domain receives the corresponding gradient (as in routed Mix-LoRA training), but Algorithm 1 should state this explicitly. The revision should specify that each expert E_k is trained only on D_k, or else the 'domain experts are never shared' argument is not well-defined.
  3. [Limitations, Appendix A] The Limitations section acknowledges the small model size, the single benchmark, and the limited set of attacks, but it does not acknowledge the mismatch between the claimed full-document DP and the actually provided per-client cross-domain guarantee. Since the formal privacy scope is the main correctness issue in the paper, the limitations discussion should explicitly state that the guarantee is per receiving domain and does not protect a domain's own documents from a client holding that domain's expert.
minor comments (6)
  1. [Appendix C.4, Figure 5 caption] The caption of Figure 5 refers to 'results in Table 5,' but the displayed MIA results are in Table 4; Table 5 is the ablation study.
  2. [Equation (1), Section 3] Equation (1) uses W on both sides of the decomposition W = W + α Σ B^(k)A^(k); renaming the adapted weight (e.g., W') would avoid notational collision.
  3. [Algorithm 1, Stage 2] Even if the intended per-domain update is clear from context, the pseudocode line Et ← Et−1 − SGD(θ0, P_T, E_{t−1}, B_t) should be rewritten to show that for a batch containing documents from domain k, only E_k is updated and other experts receive zero gradient.
  4. [Table 2 header] The column header 'PEFTPython' appears to be a formatting error; the headers for 'PEFT' and the per-domain accuracies should be separated.
  5. [Table 4 and Appendix C.4] The text should state explicitly which non-private model is used as the 'non-private result' in Table 4; Appendix C.4 mentions that the studied Mix-LoRA model is trained with SGD, but Table 4 itself does not name the baseline.
  6. [References] The bibliography entry 'Chen et al. Evaluating large language models trained on code' should use the full author list rather than 'Chen et al.' as the author name.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the privacy-claim overbreadth is a correctness concern, not a circular reduction.

full rationale

The derivation chain in NoEsis is self-contained: the knowledge-transfer claim ('bridges at least 77% of the accuracy gap') is computed from independent CodeXGLUE test sets and external baselines (Share Nothing and the non-private variant), not from any parameter fitted inside the method; the privacy accounting uses the external Opacus implementation; and the empirical MIA evaluation uses a standard log-likelihood attack on held-out data. The only self-citation (Chang et al., 2024, which shares authors Shamsabadi and Haddadi) is used to motivate a standard attack and is not load-bearing: no constant, uniqueness claim, or privacy bound is imported from it. A reviewer concern remains outside circularity: the paper states document-level DP for the full dataset (Section 3) while Algorithm 1 releases non-private per-domain LoRA experts, so the unqualified 'provable privacy guarantees' phrasing is broader than the proved per-client cross-domain guarantee. That is a rigor or overclaim issue, not a reduction of a result to its own inputs, so it does not raise the circularity score.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central result is empirical. It depends on several hand-selected hyperparameters (prompt count, LoRA rank, clipping norm, epsilon, learning rate) and on three unproved assumptions about DP accounting, block sampling, and post-processing. No new entities are introduced.

free parameters (6)
  • Number of shared prompt tokens npt = 32 (pt32)
    Selected from {8, 32, 120} in Table 3 because pt32 has the highest accuracy across Python, Java, and Go.
  • Domain LoRA rank r = 512
    Chosen because higher ranks give better results and rank 512 fits GPU memory; not tied to a formal criterion.
  • Gradient clipping norm C = 1.0
    Selected from a sweep over [0.01, 10]; a value of 0.1 gives slightly worse accuracy.
  • Privacy budget epsilon = 1.0
    Selected from {0.01, 0.1, 1.0, 8.0, 16.0} based on utility not degrading by more than one accuracy point.
  • Stage 2 learning rate = 1e-3
    Selected from {1e-4, 1e-3, 5e-3}; 1e-3 gives the highest accuracy in Table 7.
  • Common LoRA rank rc (alternative variant) = 4
    Selected using Table 3 for the common-LoRA knowledge-sharing variant.
assumptions (4)
  • standard math Opacus privacy accountant correctly computes the noise multiplier for the DP-SGD Stage 1 under the paper's sampling scheme.
    The paper uses Opacus (Yousefpour et al., 2021) without deriving the accounting, so the guarantee inherits the accountant's assumptions.
  • domain assumption Sampling one 512-token block per document per epoch preserves the document-level DP guarantee with the sampling amplification assumed by the accountant.
    Documents are arbitrary-length; the paper asserts an epoch uses each document exactly once but does not prove the block-sampling process satisfies the accountant's sampling model.
  • ad hoc to paper Stage 2 non-private expert training is post-processing of the DP-trained prompts and does not break cross-domain privacy.
    No formal composition proof is given; the argument is architectural, namely that experts are domain-local and only prompts are shared.
  • domain assumption The CodeXGLUE, PY150, JavaCorpus, and CodeSearchNet datasets are deduplicated against CodeT5+ pretraining, so fine-tuning data are not in the pretraining set.
    The paper relies on Wang et al. (2023) for deduplication to simulate private domains; if deduplication is incomplete, the privacy evaluation may be optimistic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptation." pith.science (2026). https://pith.science/paper/YHW5W7B4

@misc{pith2026250418147,
  author       = {Pith},
  title        = {Pith review of: NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YHW5W7B4}},
  note         = {Machine review of arXiv:2504.18147}
}
read the original abstract

Large Language Models (LLM) are typically trained on vast amounts of data from various sources. Even when designed modularly (e.g., Mixture-of-Experts), LLMs can leak privacy on their sources. Conversely, training such models in isolation arguably prohibits generalization. To this end, we propose a framework, NoEsis, which builds upon the desired properties of modularity, privacy, and knowledge transfer. NoEsis integrates differential privacy with a hybrid two-staged parameter-efficient fine-tuning that combines domain-specific low-rank adapters, acting as experts, with common prompt tokens, acting as a knowledge-sharing backbone. Results from our evaluation on CodeXGLUE showcase that NoEsis can achieve provable privacy guarantees with tangible knowledge transfer across domains, and empirically show protection against Membership Inference Attacks. Finally, on code completion tasks, NoEsis bridges at least 77% of the accuracy gap between the non-shared and the non-private baseline.

Figures

Figures reproduced from arXiv: 2504.18147 by the authors.

Figure 1
Figure 1. The training and the deployment process of N [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Between the results of a non￾shared model, which is the baseline, and a non-private model, which obtains the highest accuracy, NOESIS bridges the accuracy gap by more than 77%. et al., 2019); iv) Single Common Adapter: A common LoRA trained with DP across domains (Yu et al., 2021b), which is to compare the influence of using PEFT instead of monolithic fine-tuning; v) Prompt-Tuning Only: tuning only the prompts under… view at source ↗
Figure 3
Figure 3. Histograms of number of tokens per document for the three domains. The distributions [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Illustration of different baselines. Grey represents frozen, while blue, green, and red [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Visualization of results in Table 5 with all combinations of source and target domains. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 7
Figure 7. Figure 7: Illustrating the stochasticity due to training NOESIS with DP-SGD. Python, having the most data, shows the least variability, while Go, as the scarce domain, exhibits greater variability in accuracy. D.4 QUALITATIVE EXAMPLES OF MISPREDICTIONS In this analysis section, …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 25 canonical work pages

  1. [1]

    The 17 Published as a workshop paper at ICLR 2025 training loss is the cross-entropy log-likelihood averaged along the sequence and averaged among all three domains

    in the top row and for expert parameters Et in the bottom row. The 17 Published as a workshop paper at ICLR 2025 training loss is the cross-entropy log-likelihood averaged along the sequence and averaged among all three domains. D.3 R ANDOMNESS IN DP TRAINING To explore the stochasticity introduced by training with a differentially private algorithm, we r...

    work page 2025

  2. [2]

    5, we provide the ROC-AUC curves for all the combinations of Python, Java, and Go source and target languages in Fig

    Extending the results of Fig. 5, we provide the ROC-AUC curves for all the combinations of Python, Java, and Go source and target languages in Fig. 5 below. We witness that for N OESIS, the TPR is closer to the random chance line, which is when the TPR equals the the FPR. 16 Published as a workshop paper at ICLR 2025 0 20 40 60 80 100 True Positive Rate (...

    work page 2025

  3. [3]

    All results in this table are obtained under (ε = 1.0,δ = 10−6)-DP. For the domain-specific parameters, in CodeT5+ , there is a bottleneck structure inW ofW dmodel×dff i andW dff×dmodel o , wheredmodel is the model hidden size (dmodel = 768,dff = 2048 for CodeT5+ ) and we have one LoRA for each matrix. C.3 N OESIS HYPERPARAMETER DETAILS Batch Size. Keepin...

    work page

  4. [5]

    document

    First off, we ob- serve that removing any of the compo- nents of N OESIS is detrimental to its downstream accuracy, with a minimum drop of 36.89% points and an average of 50% for common prompts and 30% for domain experts. The common prompt seems to be more important in the downstream accuracy, but this can be an effect of our train- ing stage ordering. Ho...

    work page 2016

  5. [6]

    Accessed: 2024-11-11

    URL https://eur-lex.europa.eu/ legal-content/EN/TXT/?uri=CELEX%3A32024R1689. Accessed: 2024-11-11. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research,

    work page 2024

  6. [7]

    Mixture-of-loras: An effi- cient multitask tuning method for large language models

    7 Published as a workshop paper at ICLR 2025 Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-of-loras: An effi- cient multitask tuning method for large language models. InProceedings of the Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING. ELRA and ICCL,

    work page 2025

  7. [8]

    Accessed: 2024- 12-05

    URL https://www.reuters.com/technology/ artificial-intelligence/apple-delay-launch-ai-powered\ -features-europe-blames-eu-tech-rules\-2024-06-21/ . Accessed: 2024- 12-05. GitHub. Github copilot,

    work page 2024

  8. [9]

    Ac- cessed: 2024-11-11

    URL https://github.com/features/copilot. Ac- cessed: 2024-11-11. Aditya Golatkar, Alessandro Achille, Yu-Xiang Wang, Aaron Roth, Michael Kearns, and Stefano Soatto. Mixed differential privacy in computer vision. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR),

    work page 2024

Show all 40 references
  1. [10]

    Multi- coder: Multi-programming-lingual pre-training for low-resource code completion

    Zi Gong, Yinpeng Guo, Pingyi Zhou, Cuiyun Gao, Yasheng Wang, and Zenglin Xu. Multi- coder: Multi-programming-lingual pre-training for low-resource code completion. arXiv preprint 2212.09666,

    arXiv

  2. [11]

    Code- searchnet challenge: Evaluating the state of semantic code search

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Code- searchnet challenge: Evaluating the state of semantic code search. arXiv preprint 1909.09436,

    1909 arXiv

  3. [12]

    Worldwide federated training of language models

    Alex Iacob, Lorenzo Sani, Bill Marino, Preslav Aleksandrov, William F Shen, and Nicholas Donald Lane. Worldwide federated training of language models. arXiv preprint 2405.14446,

    arXiv

  4. [13]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2001.08361,

    2001 arXiv

  5. [15]

    Hints-in-browser: Benchmarking language models for programming feedback generation

    Nachiket Kotalwar, Alkis Gotovos, and Adish Singla. Hints-in-browser: Benchmarking language models for programming feedback generation. arXiv preprint 2406.05053,

    arXiv

  6. [16]

    Gshard: Scaling giant models with conditional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint 2006.16668,

    2006 arXiv

  7. [18]

    Codexglue: A machine learning benchmark dataset for code understanding and generation

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint 2102.04664,

    arXiv

  8. [21]

    Accessed: 2025-01-26

    URL https://www.salesforce.com/uk/news/ press-releases/2024/12/17/agentforce-2-0-announcement/ . Accessed: 2025-01-26. Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, and Lifu Huang. Multimodal instruction tuning with conditional mixture of lora. In Proceedings of th...

    2024

  9. [22]

    Accessed: 2024-12-05

    URL https://about.fb.com/ news/2024/06/building-ai-technology-for-europeans\ -in-a-transparent-and-responsible-way/ . Accessed: 2024-12-05. Pierre Tholoniat, Huseyin A Inan, Janardhan Kulkarni, and Robert Sim. Differentially private train- ing of mixture of experts models. AAA...

    2024

  10. [23]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2302.13971,

    arXiv

  11. [24]

    Opacus: User-friendly differential privacy library in pytorch

    Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in pytorch. arXiv preprint 2109.12298,

    arXiv

  12. [25]

    Moe-lpr: Multilingual extension of large language models through mixture- of-experts with language priors routing

    Hao Zhou, Zhijun Wang, Shujian Huang, Xin Huang, Xue Han, Junlan Feng, Chao Deng, Weihua Luo, and Jiajun Chen. Moe-lpr: Multilingual extension of large language models through mixture- of-experts with language priors routing. arXiv preprint 2408.11396,

    arXiv

  13. [26]

    Universal and transferable adversarial attacks on aligned language models

    10 Published as a workshop paper at ICLR 2025 Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint 2307.15043 ,

    2025 arXiv

  14. [27]

    However, this is an important problem with many additional dimensions to consider

    11 Published as a workshop paper at ICLR 2025 A L IMITATIONS & F UTURE WORK We have laid the foundations for a new, privacy-friendly, modular method of addressing domain pri- vacy in LLMs. However, this is an important problem with many additional dimensions to consider. The a...

    2025

  15. [28]

    We aim to explore such alternative forms for parameter-efficient adapters and their interplay with NoE in future work

    and VeRA (Kopiczko et al., 2024). We aim to explore such alternative forms for parameter-efficient adapters and their interplay with NoE in future work. There is also the question of achieving modularity, privacy, and knowledge transfer in a single-stage training method, inste...

    2024

  16. [29]

    While not in the direct scope of our study, we do acknowledge their importance in various deployment settings and future avenues of work

    and data reconstruction (Carlini et al., 2019)). While not in the direct scope of our study, we do acknowledge their importance in various deployment settings and future avenues of work. B A BLATION STUDY Table 5: Ablation: Surgically removing the shared pa- rameters, after th...

    2019

  17. [31]

    Domain Source Description Training Set Evaluation Set Pre-training GitHub Code (CodeParrot,

    The original pub- 12 Published as a workshop paper at ICLR 2025 Table 6: An overview of datasets and domains used for the training and evaluation of N OESIS on code completion tasks. Domain Source Description Training Set Evaluation Set Pre-training GitHub Code (CodeParrot,

    2025

  18. [32]

    document

    Extracted from CodeSearchNet† 2,000 1 ,559 † Dataset explicitly deduplicated in pre-training dataset (Wang et al., 2023). lishers of the data set did select for repositories with permissive licenses. While originally created for code parsing, we use only the actual code tokens...

    2023

  19. [34]

    shared parameters

    The data set for Go consists of 167,288 search queries originating from 3,559 repositories. We split the data set into 2,000 repositories for training and 1,559 repositories for testing. Functions are concatenated with interleaved linebreaks. This mimics the provisioning strat...

    2023

  20. [36]

    A small sweep of learning rates in Table 7 shows that either a smaller or larger learning rate results in lower accuracy

    with a learn- ing rate 10−3, similar to prior work (Wang et al., 2023). A small sweep of learning rates in Table 7 shows that either a smaller or larger learning rate results in lower accuracy. For scheduling, we use a linear step scheduler with 500 steps of warmup. Rank Size ...

    2011

  21. [38]

    The dataset is augmented with domain labels, which are used for deterministic routing

    The algorithm starts by computing the noise multiplierσ using the privacy account- ing mechanism (Yousefpour et al., 2021). The dataset is augmented with domain labels, which are used for deterministic routing. In each iteration, a batch of documents is ran- domly drawn from a...

    2021

  22. [40]

    Setting [

    The bottom row illustrates the second training stage. While the training losses appear similar, training with 32 prompt tokens achieves the highest test accuracy. 69.13 69.14 69.15 Python 61.17 61.18 61.19 61.20 Java 66.52 66.54 66.56 66.58 66.60 Go Figure 7: Illustrating the ...

    2025

  23. [2013]

    below average

    The original publishers of the data set did select for repositories with permissive licences and repositories that were forked at least once. Additionally, repositories with a “below average” reputation score on GitHub are not considered. In 2021, the CODEXGLUE effort repeated...

    2021

  24. [2014]

    Regulation (eu) 2024/1689,

    EU-regulation. Regulation (eu) 2024/1689,

    2024

  25. [2016]

    Peters, Swabha Swayamdipta, and Thomas Wolf

    9 Published as a workshop paper at ICLR 2025 Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta, and Thomas Wolf. Transfer learning in natural language processing. In Anoop Sarkar and Michael Strube (eds.), Proceedings of the Annual Meeting of the Association for Computati...

    2025

  26. [2019]

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang

    ISBN 9781787781320. Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprint 2407.06204,

    arXiv

  27. [2020]

    Sparse upcycling: Training mixture-of-experts from dense checkpoints

    8 Published as a workshop paper at ICLR 2025 Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints. arXiv preprin...

    2025 arXiv

  28. [2021]

    Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts

    Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. arXiv preprint 2404.15159,

    arXiv

  29. [2022]

    Context-aware membership inference attacks against pre-trained large language models

    Hongyan Chang, Ali Shahin Shamsabadi, Kleomenis Katevas, Hamed Haddadi, and Reza Shokri. Context-aware membership inference attacks against pre-trained large language models. arXiv preprint 2409.13745,

  30. [2023]

    com/index/gpt-3-5-turbo-fine-tuning-and-api-updates/

    URL https://openai. com/index/gpt-3-5-turbo-fine-tuning-and-api-updates/ . Accessed: 2025- 01-26. F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Bruch...

    2025

  31. [2024]

    Evaluating large language models trained on code

    Chen et al. Evaluating large language models trained on code. arXiv preprint 2107.03374,

    arXiv

  32. [2025]

    Accessed: 2025-01-09

    URL https://huggingface.co/datasets/ codeparrot/github-code. Accessed: 2025-01-09. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the North Ameri- can Chapter...

    2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.