Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks

Bingjie Yan; Jianbing Ding; Jianqing Zhang; Jingyi Li; Kejia Zhang; Qiang Yang; Tianyuan Zou; Xiangsen Chen; Xiaozhou Ye; Xidong Wang

REVIEW 4 major objections 4 minor 1 cited by

Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that private-domain AI should be built by pairing large general-purpose models with small on-site models, moving only distilled knowledge across the boundary instead of raw data or full model weights.

desk verdict A serviceable survey/position paper with a sensible taxonomy; the central advocacy is a research agenda rather than a demonstrated result, and the paper is honest about that. read the letter →

arxiv 2504.17421 v2 pith:3CBZU3OK submitted 2025-04-24 cs.LG cs.AI

Yang Liu , Kejia Zhang , Bingjie Yan , Tianyuan Zou , Jianqing Zhang , Zixuan Gu , Xiangsen Chen , Jianbing Ding

show 6 more authors

Xidong Wang Jingyi Li Xiaozhou Ye Ye Ouyang Qiang Yang Ya-Qin Zhang

This is my paper · ORCID

classification cs.LGcs.AI

keywords largelanguagemodelssmallknowledgedistillationfederatedlearningdataprivacyprivate-domainadaptationcross-silocollaborationmulti-objectivebenchmarking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that private-domain AI should be built by pairing a large general-purpose model with small on-site models rather than choosing between them. Its thesis is that the large model's breadth and the small model's domain specialization can be exchanged as knowledge carriers—logits and representations, synthetic data, or adapters and prompts—without moving raw data or exposing proprietary weights. The authors survey the literature through three directions: downward transfer from large to small models, upward transfer from small to large models, and inference-time collaboration, and they unify these under one constrained optimization problem with privacy, model-leakage, and resource budgets. They also propose a multi-objective benchmark so that methods can be compared on utility, privacy, efficiency, and query cost together rather than accuracy alone. The paper is explicit in Section 4.3 that the field lacks standardized privacy metrics and real-world private benchmarks, so the contribution is a framing and a research agenda rather than an evaluated method.

What carries the argument

The load-bearing idea is the knowledge carrier, the compact artifact that crosses the boundary between the data owner and the model owner in place of raw data and full weights. The paper identifies three families: logits and representations, synthetic data, and adapters or prompts; each can travel downward from large to small models, upward from small to large models, or both at inference time. Carriers do the work of separating the utility objective from the privacy, model-security, and resource constraints, so the tractability of the whole paradigm depends on whether these artifacts preserve enough task-relevant information to justify the collaboration.

What would settle it

Run the best cross-silo collaboration on a genuinely private real-world dataset against three baselines—training the small model locally, fine-tuning the large model centrally on the same data, and standard federated learning—and measure utility, privacy leakage (for example, success of reconstruction or membership attacks), communication, computation, and query cost. If the collaborative method is not Pareto-better than at least one baseline on the multi-objective comparison, the paper's central advocacy is defeated. The paper itself notes the lack of standardized privacy metrics and real private benchmarks, so designing this comparison is the direct test of its thesis.

Watch

Extended reading notes

Core claim

The central claim is that large and small models are complements: the large model contributes general knowledge, generative ability, and reasoning; the small model contributes private-domain specialization, on-site efficiency, and data access that the large model is forbidden to see. Knowledge can flow in both directions—distilled logits or representations, synthetic datasets, and lightweight adapters or prompts are the carriers—so a domain task can be solved without centralizing data and without releasing the large model's parameters. The paper formalizes this as a constrained optimization: maximize task utility while keeping privacy leakage, model leakage, and local computation within prescribed budgets. Its taxonomy claims that every existing approach fits one of three directions: downward transfer, upward transfer, or inference-time collaboration, with split learning, collaborative decoding, and retrieval as the inference-time family. The authors position this as the pattern for adapting large models to private domains and call for application-driven, multi-objective evaluation on real private data as the way to test it.

Load-bearing premise

The enabling premise is that the small carriers—logits, synthetic samples, prototypes, adapters, and prompts—can carry enough task-relevant knowledge across a privacy and security boundary while genuinely satisfying privacy, model-IP, and resource constraints; the paper surveys techniques but does not demonstrate this.

Editorial extensions

If this is right

Domain adaptation would no longer require shipping raw private data or full model weights; knowledge could travel as distilled predictions, synthetic samples, prototypes, or trained adapters.
Comparisons of collaboration strategies should be framed by the constrained problem with privacy, model-leakage, and resource budgets rather than by accuracy alone.
Benchmarks on open datasets such as IMDB overstate real performance because the domain gap to true private data is minimal, so private datasets that report the domain gap are needed.
A customizable multi-objective scoring function can rank methods on utility, privacy, communication, computation, and large-model query cost at once.
Inference-time collaboration extends the same privacy-respecting knowledge flow to settings where fine-tuning either model is infeasible, using split learning, collaborative decoding, or retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural experiment the paper does not run: hold one large model fixed and compare the three carriers—logits, synthetic data, and adapters—on the same private task under the same privacy budget; the result would show which carrier actually deserves investment.
If strong privacy guarantees force the carriers to be heavily sanitized, the argument implicitly caps the paradigm's ceiling at the small model's domain coverage, a consequence the authors do not spell out.
The proposed multi-objective benchmark could be instantiated as a Pareto frontier over utility, privacy leakage, communication cost, and query cost; leaving the scoring function unspecified is reasonable for a position paper, but a concrete instantiation would make the framework testable.
The paper mentions one industrial deployment as evidence of feasibility; a public case study reporting utility, privacy, and cost numbers on that system would convert the position into evidence.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a look if you work in federated or privacy-preserving LLM adaptation. The real contribution is organizational: the paper groups the scattered literature on large-small model collaboration into three directions—downward transfer, upward transfer, and inference-time collaboration—and ties them to a constrained optimization view (Eq. 1) that makes the privacy/security/efficiency trade-offs explicit. That framing is useful and mostly faithful to the cited work. I don't see derivation errors because there are no derivations; the equations are definitions, and the paper doesn't claim otherwise.

What it does well: the taxonomy is sensible, the coverage is broad, and Section 4 does a decent job cataloging defenses (DP, adversarial training, unlearning, sanitization, secure computation) and efficiency considerations. The paper is also honest about the field's gaps: Section 4.3 concedes that privacy metrics are not standardized and real-world private benchmarks are missing. That honesty is a point in its favor.

Where it wobbles: the "comprehensive overview" claim in the abstract oversells what is essentially a curated, narrative survey—there is no systematic search or inclusion protocol, so coverage is likely biased toward the authors' own network. The paper also leans on the authors' own methods (FuseGen, FedTGP, CrossLM, split-and-privatize) when motivating categories; that's not disqualifying, but a reader should be aware. The bigger soft spot is the load-bearing advocacy: the claim that LM-SM collaboration can accelerate private-domain adaptation is asserted, not demonstrated. The paper cites real privacy breaks (paired-logits inversion, gradient inversion, RAG leaks) against the very carriers it advocates, then concedes we don't yet have metrics to check whether the trade-off works. That's a fair state of the field, but it means the paper is a research agenda, not a validated approach. The one anecdotal deployment mention (AsiaInfo) is too thin to count as evidence.

Who it's for: newcomers to federated or privacy-preserving LLM adaptation who want a map. Researchers already in the area won't find much new. With revisions to tone down the comprehensiveness claim and separate the taxonomy from the advocacy, it would be a solid survey. As is, it deserves a serious referee but not unconditional acceptance.

Referee Report

4 major / 4 minor

Summary. This position paper argues that collaboration between large language models (LMs) and small models (SMs) can accelerate domain adaptation in private, cross-boundary settings where data, model parameters, and resources are constrained. The paper formalizes the collaboration as a constrained optimization problem in Eq. (1), proposes a taxonomy of three research directions (knowledge transfer from LMs to SMs, from SMs to LMs, and inference-time collaboration), and surveys representative techniques in each direction. It then discusses privacy threats, efficiency considerations, and benchmarking gaps, and proposes a flexible multi-objective evaluation framework in Eq. (2). The paper closes with a call for application-driven research and real-world private benchmarks, including an anecdotal example of a deployed industrial system.

Significance. If the central thesis is accepted, the paper provides a useful organizing framework for a rapidly growing and fragmented literature, connecting model collaboration to privacy, security, and efficiency constraints in a way that many prior surveys do not. The problem formulation in Eq. (1) and the proposed taxonomy could serve as a shared vocabulary for future work. The paper also usefully highlights the lack of standardized privacy metrics and real-world private benchmarks, which is an important gap. However, the paper makes no quantitative claims and does not establish its central enabling condition: that the surveyed knowledge carriers (logits, representations, synthetic data, adapters, prompts) can transfer sufficient task-relevant information while satisfying the constraints in Eq. (1). The proposed benchmark in Eq. (2) is not yet operational, and the survey's "comprehensive" coverage is not supported by a systematic methodology. These issues limit the paper's current contribution to a roadmap rather than a validated framework.

major comments (4)

[§4.3 and §4.1] The paper's central advocacy, stated in the abstract and Section 1, is that LM–SM collaboration can accelerate private-domain adaptation. This claim depends on the premise that the information carriers surveyed in §3.4.1–3.4.4 can transfer enough task-relevant signal across the silo boundary while keeping privacy leakage, model leakage, and resource costs within the bounds of Eq. (1). The paper never demonstrates this premise. Section 4.1 cites concrete privacy breaks against these very carriers (e.g., paired-logits inversion for FedMD in [155], gradient inversion in [153], RAG privacy risks in [149]), and Section 4.3 concedes that "quantifying the privacy implications of emerging privacy-preserving techniques using DP remains an open challenge" and that real-world private benchmarks are absent. Eq. (1) therefore remains a template rather than a checkable feasibility problem. The authors should either synthesize the available positive and negative empirical evidence on whether these carriers can operate within the constraints, or explicitly reframe the central claim as a hypothesis that motivates a research agenda.
[§5.1, Eq. (2)] The proposed multi-objective benchmark defines a scoring function f that is "flexible" and can be "carefully designed," but no concrete instantiation is given. As written, f can be chosen to rank any method arbitrarily, so the benchmark is not falsifiable. To be usable, the paper should propose at least one default f (e.g., a weighted linear combination), specify a protocol for reporting the individual objectives o_1, ..., o_n, and state how trade-offs are to be handled without post hoc reweighting.
[§3.4, Table 1] The paper claims that its taxonomy classifies the literature into three complete directions, but no systematic methodology—such as a search protocol, inclusion/exclusion criteria, or coverage statistics—is provided to support the "comprehensive" claim. In addition, the taxonomy contains internal inconsistencies: Table 1 lists collaborative decoding as transferring "insensitive data" from S to L, whereas the text in §3.4.4 states that "these approaches directly expose private data to LMs during inference time," with CoGenesis [143] being the exception. This inconsistency undermines the reliability of the taxonomy as a classification of the field.
[§5.2] The paper cites as evidence a deployed AsiaInfo system for wireless network complaints, claiming that "test results show that it not only significantly enhanced user experience but also greatly improved the efficiency of conducting data analysis." No data, evaluation protocol, or metrics are provided. If this deployment is intended to support the central thesis, the paper should either include concrete evaluation results or clearly mark the passage as an anecdotal illustration rather than evidence.

minor comments (4)

[§2] There is a missing space in "utilizeSmall Models" in the third paragraph of Section 2.
[References] References [1] and [19] appear to cite the same work (Villalobos et al., "Will we run out of data?") in two versions; this should be consolidated or cross-referenced.
[References] Reference [219] contains a typo: "Syste m design" should be "System design."
[§3.4] The phrase "Table 1 and Figure 2 summarized" should be "Table 1 and Figure 2 summarize" for subject-verb agreement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the survey's taxonomy and framework are independent of its inputs; self-citations are illustrative, and the conceded metric gaps are correctness issues, not circular steps.

full rationale

The paper is a survey/position paper rather than a derivation with fitted parameters or predicted quantities. Its central claim—that LM–SM collaboration can accelerate private-domain adaptation—is an advocacy assertion supported by a broad literature, and no equation in the paper reduces to its own inputs. Equation (1) is a deliberately general problem template whose constraints (Mp, ML, Me) are never instantiated, and Eq. (2) is an explicitly customizable objective function, so neither can constitute a self-definitional or fitted-input circularity. The taxonomy is descriptive: distillation-, generation-, and parameter-based transfer are established categories in the cited literature, and the authors' own works (CrossLM, FuseGen, FedTGP, split-and-privatize) appear only as examples within categories that also contain many external references. These self-citations recur, but the load-bearing organization and motivation do not rest on them alone. The paper's own limitations in Section 4.3—for example, 'quantifying the privacy implications of emerging privacy-preserving techniques using DP remains an open challenge' and the absence of real-world private benchmarks—identify an unverified enabling condition for the advocacy, which is a correctness and feasibility gap rather than a circular step. No prediction is fitted, no uniqueness theorem is imported from the authors' prior work, and no known result is merely renamed as a new contribution. Therefore no circularity is found.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This ledger is minimal because the paper is a survey. It introduces no fitted parameters or invented entities. The constraints epsilon_p, epsilon_L, and epsilon_e in Eq. (1) are practitioner-chosen budgets, not quantities fitted to data. The axioms above capture the boundary model, the assumed completeness of the taxonomy, and the unvalidated benchmark premise.

assumptions (4)

domain assumption Private domain data cannot be shared with the LM party, and local hardware can only run small models; LM parameters cannot be shared due to ownership and resource constraints.
This is the boundary model the whole survey is built around, stated in Section 2's problem definition and Figure 1.
ad hoc to paper The three knowledge-carrier families (logits and representations, synthetic data, adapters and prompts) are sufficient to cover the existing design space.
The taxonomy's completeness is asserted in Section 3.4 and Table 1 rather than derived from a systematic search.
domain assumption Privacy, model leakage, and efficiency can be measured by Mp, Me, and ML and meaningfully constrained in a single optimization.
Eq. (1) assumes these constraints exist and are comparable, although Section 4.3 notes that privacy metrics are not standardized.
ad hoc to paper A real-world multi-objective benchmark on private data can be constructed with a scoring function f.
Eq. (2) proposes this benchmark, but no dataset or implementation is provided; Section 4.3 lists it as an open challenge.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks." pith.science (2026). https://pith.science/paper/3CBZU3OK

@misc{pith2026250417421,
  author       = {Pith},
  title        = {Pith review of: Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3CBZU3OK}},
  note         = {Machine review of arXiv:2504.17421}
}

read the original abstract

Large language models (LMs) offer broad generalization capabilities but require vast amounts of data and computational resources for domain-specific tasks; small models (SMs), in contrast, are more efficient and tailored to specific domains yet lack general-purpose coverage. Taking a collaborative approach, where large and small models work synergistically, can accelerate the adaptation of LLMs to private domains and unlock new potential in AI. This survey presents a comprehensive overview of recent advances and challenges in harnessing the collaborative power of large and small models for private-domain adaptation. It specifically focuses on the unique constraints of cross-boundary environments, where models belong to distinct parties, and examines the resulting tensions among data privacy, model security, integrity, and resource limitations. By analyzing the information flow between distinct model and data stakeholders, we propose a unified taxonomy that classifies research into three primary directions: downward knowledge transfer (LM to SM), upward knowledge transfer (SM to LM), and inference-time collaboration across parties. Drawing on this taxonomy, we analyze the core challenges inherent to cross-boundary information exchange, including data-privacy, model-security, and integrity threats as well as efficiency constraints, and synthesize these into a multi-objective optimization problem that governs practical deployment. Finally, we review key open challenges inherent to such hybrid approaches and outline promising directions for future research. By offering a principled, boundary-centric view of this rapidly evolving landscape, this survey aims to serve as a structured resource for researchers and practitioners advancing privacy-aware, resource-efficient AI deployment.

Figures

Figures reproduced from arXiv: 2504.17421 by the authors.

**Figure 2.** Collaboration of LMs and SMs for domain tasks: overview of techniques. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
cs.DC 2025-07 conditional novelty 4.0 of 10

A survey that builds a taxonomy of edge-cloud LLM-SLM collaboration for inference and training, claiming to be the first to unify both phases.

Reference graph

Works this paper leans on

221 extracted references · 26 canonical work pages · cited by 1 Pith paper

[155]

Breaching fedmd: Image recovery via paired-logits inversion attack

Hideaki Takahashi, Jingjing Liu, and Yang Liu. Breaching fedmd: Image recovery via paired-logits inversion attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12198–12207, 2023

2023
[153]

Deep leakage from gradients

Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. dÁlché-Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Systems 32, pages 14774–14784. Curran Associates, Inc., 2019

2019
[149]

The good and the bad: Exploring privacy issues in retrieval-augmented generation (RAG)

Shenglai Zeng, Jiankun Zhang, Pengfei He, Yiding Liu, Yue Xing, Han Xu, Jie Ren, Yi Chang, Shuaiqiang Wang, Dawei Yin, and Jiliang Tang. The good and the bad: Exploring privacy issues in retrieval-augmented generation (RAG). In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, page...

2024
[143]

CoGenesis: A framework collaborating large and small language models for secure context-aware instruction following

Kaiyan Zhang, Jianyu Wang, Ermo Hua, Biqing Qi, Ning Ding, and Bowen Zhou. CoGenesis: A framework collaborating large and small language models for secure context-aware instruction following. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

2024
[1]

Will we run out of data? limits of llm scaling based on human-generated data, 2024

Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Will we run out of data? limits of llm scaling based on human-generated data, 2024

2024
[2]

Advances and open challenges in federated foundation models, 2024

Chao Ren, Han Yu, Hongyi Peng, Xiaoli Tang, Bo Zhao, Liping Yi, Alysa Ziying Tan, Yulan Gao, Anran Li, Xiaoxiao Li, Zengxiang Li, and Qiang Yang. Advances and open challenges in federated foundation models, 2024

2024
[3]

Shenghui Li, Fanghua Ye, Meng Fang, Jiaxu Zhao, Yun-Hin Chan, Edith C. H. Ngai, and Thiemo V oigt. Synergizing foundation models and federated learning: A survey, 2024

2024
[4]

Introducing chatgpt

OpenAI. Introducing chatgpt. OpenAI, 2022

2022

Show all 221 references

[5]

Gpt-4 technical report

OpenAI. Gpt-4 technical report. OpenAI, 2023

2023
[6]

A survey of resource-efficient llm and multimodal foundation models

Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, et al. A survey of resource-efficient llm and multimodal foundation models. arXiv preprint arXiv:2401.08092, 2024

2024 arXiv
[7]

Bloomberggpt: A large language model for finance

Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023

2023 arXiv
[8]

Deepseek-v3 technical report, 2024

DeepSeek-AI. Deepseek-v3 technical report, 2024

2024
[9]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

2016
[10]

Long short-term memory

S Hochreiter. Long short-term memory. Neural Computation MIT-Press, 1997

1997
[11]

Empirical evaluation of gated recurrent neural networks on sequence modeling

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014

2014 arXiv
[12]

Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S

Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jare...

2022
[13]

Llava-med: Training a large language-and-vision assistant for biomedicine in one day

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890, 2023

2023 arXiv
[14]

Biogpt: generative pre-trained transformer for biomedical text generation and mining

Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in bioinformatics, 23(6):bbac409, 2022

2022
[15]

Biobert: a pre-trained biomedical language representation model for biomedical text mining

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 2020

2020
[16]

Phi-2: The surprising power of small language models

Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023

2023
[17]

Phi-3 technical report: A highly capable language model locally on your phone

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

2024 arXiv
[18]

Introducing llama 3.1: Our most capable models to date

MetaAI. Introducing llama 3.1: Our most capable models to date. MetaAI, 2024

2024
[19]

Position: Will we run out of data? limits of llm scaling based on human-generated data

Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: Will we run out of data? limits of llm scaling based on human-generated data. In Forty-first International Conference on Machine Learning
[20]

Communication- efficient learning of deep networks from decentralized data

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273–1282. PMLR, 2017

2017
[21]

Federated machine learning: Concept and applications

Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1–19, 2019

2019
[22]

Regulation (eu) 2016/679 of the european parliament and of the council

Protection Regulation. Regulation (eu) 2016/679 of the european parliament and of the council. Regulation (eu), 679:2016, 2016

2016
[23]

California consumer privacy act (ccpa)

Rob Bonta. California consumer privacy act (ccpa). Retrieved from State of California Department of Justice: https://oag. ca. gov/privacy/ccpa, 2022

2022
[24]

Health insurance portability and accountability act of 1996

Accountability Act. Health insurance portability and accountability act of 1996. Public law, 104:191, 1996

1996
[25]

Federated learning for healthcare domain-pipeline, applications and challenges

Madhura Joshi, Ankit Pal, and Malaikannan Sankarasubbu. Federated learning for healthcare domain-pipeline, applications and challenges. ACM Transactions on Computing for Healthcare, 3(4):1–36, 2022

2022
[26]

Review on security of federated learning and its application in healthcare

Hao Li, Chengcheng Li, Jian Wang, Aimin Yang, Zezhong Ma, Zunqian Zhang, and Dianbo Hua. Review on security of federated learning and its application in healthcare. Future Generation Computer Systems , 144:271–290, 2023

2023
[27]

Deep learning-based classification of mesothelioma improves prediction of patient outcome

Pierre Courtiol, Charles Maussion, Matahi Moarii, Elodie Pronier, Samuel Pilcer, Meriem Sefta, Pierre Manceron, Sylvain Toldo, Mikhail Zaslavskiy, Nolwenn Le Stang, et al. Deep learning-based classification of mesothelioma improves prediction of patient outcome. Nature medicin...

2019
[28]

Use of federated learning and blockchain towards securing financial services

Pushpita Chatterjee, Debashis Das, and Danda B Rawat. Use of federated learning and blockchain towards securing financial services. arXiv preprint arXiv:2303.12944, 2023

2023 arXiv
[29]

Efficient and secure federated learning for financial applications

Tao Liu, Zhi Wang, Hui He, Wei Shi, Liangliang Lin, Ran An, and Chenhao Li. Efficient and secure federated learning for financial applications. Applied Sciences, 13(10):5877, 2023

2023
[30]

Machine learning ledger orchestration for drug discovery, 2019

E Cordis. Machine learning ledger orchestration for drug discovery, 2019

2019
[31]

Protecting intellectual property of large language model-based code generation apis via watermarks

Zongjie Li, Chaozheng Wang, Shuai Wang, and Cuiyun Gao. Protecting intellectual property of large language model-based code generation apis via watermarks. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 2336–2350, 2023

2023
[32]

History, development, and principles of large language models-an introductory survey

Zhibo Chu, Shiwen Ni, Zichong Wang, Xi Feng, Chengming Li, Xiping Hu, Ruifeng Xu, Min Yang, and Wenbin Zhang. History, development, and principles of large language models-an introductory survey. arXiv preprint arXiv:2402.06853, 2024

2024 arXiv
[33]

Llmcarbon: Modeling the end-to-end carbon footprint of large language models, 2024

Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi, Prateek Sharma, Fan Chen, and Lei Jiang. Llmcarbon: Modeling the end-to-end carbon footprint of large language models, 2024. 11 A PREPRINT - A PRIL 25, 2025

2024
[34]

A comprehensive survey on pretrained foundation models: A history from bert to chatgpt

Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419, 2023

2023 arXiv
[35]

Distilling the knowledge in a neural network

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

2015 arXiv
[36]

Knowledge distillation: A survey

Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. Interna- tional Journal of Computer Vision, 129(6):1789–1819, 2021

2021
[37]

A survey on knowledge distillation of large language models

Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116, 2024

2024 arXiv
[38]

Survey on knowledge distillation for large language models: Methods, evaluation, and application

Chuanpeng Yang, Yao Zhu, Wang Lu, Yidong Wang, Qian Chen, Chenlong Gao, Bingjie Yan, and Yiqiang Chen. Survey on knowledge distillation for large language models: Methods, evaluation, and application. ACM Transactions on Intelligent Systems and Technology, 2024

2024
[39]

Minillm: Knowledge distillation of large language models

Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543, 2023

2023 arXiv
[40]

On-policy distillation of language models: Learning from self-generated mistakes

Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024

2024
[41]

For distillation, tokens are not all you need

Mrigank Raman, Pranav Mani, Davis Liang, and Zachary Lipton. For distillation, tokens are not all you need. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023

2023
[42]

Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty

Inar Timiryasov and Jean-Loup Tastet. Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 279–289, 2023

2023
[43]

Llavadi: What matters for multimodal large language models distillation

Shilin Xu, Xiangtai Li, Haobo Yuan, Lu Qi, Yunhai Tong, and Ming-Hsuan Yang. Llavadi: What matters for multimodal large language models distillation. arXiv preprint arXiv:2407.19409, 2024

2024 arXiv
[44]

Less is more: Task-aware layer-wise distillation for language model compression

Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less is more: Task-aware layer-wise distillation for language model compression. In International Conference on Machine Learning , pages 20852–20867. PMLR, 2023

2023
[45]

Ddk: Distilling domain knowledge for efficient large language models

Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanxing Zhang, Haoran Que, Ken Deng, Zhiqi Bai, Jie Liu, Ge Zhang, Jiakai Wang, et al. Ddk: Distilling domain knowledge for efficient large language models. arXiv preprint arXiv:2407.16154, 2024

2024 arXiv
[46]

Boxin Wang, Yibo Zhang, Yuan Cao, Bo Li, Hugh McMahan, Sewoong Oh, Zheng Xu, and Manzil Zaheer. Can public large language models help private cross-device federated learning? In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational...

2024
[47]

Crash: Clustering, removing, and sharing enhance fine-tuning without full large language model

Kaiyan Zhang, Ning Ding, Biqing Qi, Xuekai Zhu, Xinwei Long, and Bowen Zhou. Crash: Clustering, removing, and sharing enhance fine-tuning without full large language model. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical M...

2023
[48]

Orchestration of emulator assisted mobile edge tuning for ai foundation models: A multi-agent deep reinforcement learning approach, 2023

Wenhan Yu, Terence Jie Chua, and Jun Zhao. Orchestration of emulator assisted mobile edge tuning for ai foundation models: A multi-agent deep reinforcement learning approach, 2023

2023
[49]

Offsite-tuning: Transfer learning without full model

Guangxuan Xiao, Ji Lin, and Song Han. Offsite-tuning: Transfer learning without full model. arXiv preprint arXiv:2302.04870, 2023

2023 arXiv
[50]

Fedpft: Federated proxy fine-tuning of foundation models

Zhaopeng Peng, Xiaoliang Fan, Yufan Chen, Zheng Wang, Shirui Pan, Chenglu Wen, Ruisheng Zhang, and Cheng Wang. Fedpft: Federated proxy fine-tuning of foundation models. In Kate Larson, editor, Proceedings of the Thirty-Third International Joint Conference on Artificial Intelli...

2024
[51]

Fedmd: Heterogenous federated learning via model distillation

Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019

1910 arXiv
[52]

Ensemble attention distillation for privacy-preserving federated learning

Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David Doermann, and Arun Innanje. Ensemble attention distillation for privacy-preserving federated learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15076–15086, ...

2021
[53]

Preserving privacy in federated learning with ensemble cross-domain knowledge distillation

Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David Doermann, and Arun Innanje. Preserving privacy in federated learning with ensemble cross-domain knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, p...

2022
[54]

Data shunt: Collaboration of small and large models for lower costs and better performance

Dong Chen, Yueting Zhuang, Shuo Zhang, Jinfeng Liu, Su Dong, and Siliang Tang. Data shunt: Collaboration of small and large models for lower costs and better performance. Proceedings of the AAAI Conference on Artificial Intelligence, 38(10):11249–11257, Mar. 2024

2024
[55]

Parameterized knowledge transfer for personalized federated learning

Jie Zhang, Song Guo, Xiaosong Ma, Haozhao Wang, Wenchao Xu, and Feijie Wu. Parameterized knowledge transfer for personalized federated learning. Advances in Neural Information Processing Systems, 34:10092– 10104, 2021

2021
[56]

Ensemble distillation for robust model fusion in federated learning

Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in neural information processing systems, 33:2351–2363, 2020

2020
[57]

Depth anything: Unleashing the power of large-scale unlabeled data

Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10371–10381, 2024

2024
[58]

Robust federated learning with noisy and heterogeneous clients

Xiuwen Fang and Mang Ye. Robust federated learning with noisy and heterogeneous clients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10072–10081, 2022

2022
[59]

Fedtgp: Trainable global prototypes with adaptive-margin- enhanced contrastive learning for data and model heterogeneity in federated learning

Jianqing Zhang, Yang Liu, Yang Hua, and Jian Cao. Fedtgp: Trainable global prototypes with adaptive-margin- enhanced contrastive learning for data and model heterogeneity in federated learning. Proceedings of the AAAI Conference on Artificial Intelligence, 2024

2024
[60]

An upload-efficient scheme for transferring knowledge from a server-side pre-trained generator to clients in heterogeneous federated learning

Jianqing Zhang, Yang Liu, Yang Hua, and Jian Cao. An upload-efficient scheme for transferring knowledge from a server-side pre-trained generator to clients in heterogeneous federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

2024
[61]

Learning from human educational wisdom: A student-centered knowledge distillation method.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

Shunzhi Yang, Jinfeng Yang, MengChu Zhou, Zhenhua Huang, Wei-Shi Zheng, Xiong Yang, and Jin Ren. Learning from human educational wisdom: A student-centered knowledge distillation method.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

2024
[62]

Bert learns to teach: Knowledge distillation with meta learning

Wangchunshu Zhou, Canwen Xu, and Julian McAuley. Bert learns to teach: Knowledge distillation with meta learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7037–7049, 2022

2022
[63]

Fedgems: Federated learning of larger server models via selective knowledge fusion, 2021

Sijie Cheng, Jingwen Wu, Yanghua Xiao, Yang Liu, and Yang Liu. Fedgems: Federated learning of larger server models via selective knowledge fusion, 2021

2021
[64]

Multimodal federated learning via contrastive representation ensemble

Qiying Yu, Yang Liu, Yimu Wang, Ke Xu, and Jingjing Liu. Multimodal federated learning via contrastive representation ensemble. arXiv preprint arXiv:2302.08888, 2023

2023 arXiv
[65]

Ideal: Query-efficient data-free learning from black-box models

Jie Zhang, Chen Chen, and Lingjuan Lyu. Ideal: Query-efficient data-free learning from black-box models. In The Eleventh International Conference on Learning Representations, 2022

2022
[66]

Towards data-free model stealing in a hard label setting

Sunandini Sanyal, Sravanti Addepalli, and R Venkatesh Babu. Towards data-free model stealing in a hard label setting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15284–15293, 2022

2022
[67]

Zero-shot knowledge distillation from a decision-based black-box model

Zi Wang. Zero-shot knowledge distillation from a decision-based black-box model. In International Conference on Machine Learning, pages 10675–10685. PMLR, 2021

2021
[68]

Bridging the gap between decision and logits in decision- based knowledge distillation for pre-trained language models

Qinhong Zhou, Zonghan Yang, Peng Li, and Yang Liu. Bridging the gap between decision and logits in decision- based knowledge distillation for pre-trained language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of...

2023
[69]

Synthetic data distillation enables the extraction of clinical information at scale

Elizabeth Geena Woo, Michael C Burkhart, Emily Alsentzer, and Brett Beaulieu-Jones. Synthetic data distillation enables the extraction of clinical information at scale. medRxiv, pages 2024–09, 2024

2024
[70]

ZeroGen: Efficient zero-shot learning via dataset generation

Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. ZeroGen: Efficient zero-shot learning via dataset generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11653–11669, 2022

2022
[71]

ProGen: Progressive zero- shot dataset generation via in-context feedback

Jiacheng Ye, Jiahui Gao, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. ProGen: Progressive zero- shot dataset generation via in-context feedback. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3671–3683, 2022. 13 A PREPRINT - A PRIL 25, 2025

2022
[72]

Generating training data with language models: Towards zero-shot language understanding

Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding. Advances in Neural Information Processing Systems, 35:462–477, 2022

2022
[73]

Self-guided noise-free data generation for efficient zero-shot learning

Jiahui Gao, Renjie Pi, Lin Yong, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. Self-guided noise-free data generation for efficient zero-shot learning. In The Eleventh International Conference on Learning Representations, 2023

2023
[74]

Fusegen: Plm fusion for data-generation based zero-shot learning

Tianyuan Zou, Yang Liu, Peng Li, Jianqing Zhang, Jingjing Liu, and Ya-Qin Zhang. Fusegen: Plm fusion for data-generation based zero-shot learning. arXiv preprint arXiv:2406.12527, 2024

2024 arXiv
[75]

Retrieval-based knowledge transfer: An effective approach for extreme large language model compression

Jiduan Liu, Jiahao Liu, Qifan Wang, Jingang Wang, Xunliang Cai, Dongyan Zhao, Ran Wang, and Rui Yan. Retrieval-based knowledge transfer: An effective approach for extreme large language model compression. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the A...

2023
[76]

Seeking neural nuggets: Knowledge transfer in large language models from a parametric perspective

Ming Zhong, Chenxin An, Weizhu Chen, Jiawei Han, and Pengcheng He. Seeking neural nuggets: Knowledge transfer in large language models from a parametric perspective. arXiv preprint arXiv:2310.11451, 2023

2023 arXiv
[77]

Mutual enhancement of large and small language models with cross-silo knowledge transfer, 2023

Yongheng Deng, Ziqing Qiao, Ju Ren, Yang Liu, and Yaoxue Zhang. Mutual enhancement of large and small language models with cross-silo knowledge transfer, 2023

2023
[78]

Zico Kolter

Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models, 2024

2024
[79]

Zeroquant: Efficient and affordable post-training quantization for large-scale transformers

Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168–27183, 2022

2022
[80]

Initializing models with larger ones, 2023

Zhiqiu Xu, Yanjie Chen, Kirill Vishniakov, Yida Yin, Zhiqiang Shen, Trevor Darrell, Lingjie Liu, and Zhuang Liu. Initializing models with larger ones, 2023

2023
[81]

Learning to teach with student feedback

Yitao Liu, Tianxiang Sun, Xipeng Qiu, and Xuanjing Huang. Learning to teach with student feedback. arXiv preprint arXiv:2109.04641, 2021

2021 arXiv
[82]

Meta pseudo labels

Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11557–11568, 2021

2021
[83]

Dual knowledge distillation for bidirectional neural machine translation

Huaao Zhang, Shigui Qiu, and Shilong Wu. Dual knowledge distillation for bidirectional neural machine translation. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–7. IEEE, 2021

2021
[84]

Shadow knowledge distillation: Bridging offline and online knowledge transfer

Lujun Li and Zhe Jin. Shadow knowledge distillation: Bridging offline and online knowledge transfer. Advances in Neural Information Processing Systems, 35:635–649, 2022

2022
[85]

Reverse knowledge distillation: Training a large model using a small one for retinal image matching on limited data

Sahar Almahfouz Nasser, Nihar Gupte, and Amit Sethi. Reverse knowledge distillation: Training a large model using a small one for retinal image matching on limited data. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 7778–7787, 2024

2024
[86]

Weak-to-strong generalization: Eliciting strong capabilities with weak supervision

Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeffrey Wu. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Rusla...

2024
[87]

Fedmkt: Federated mutual knowledge transfer for large and small language models

Tao Fan, Guoqiang Ma, Yan Kang, Hanlin Gu, Lixin Fan, and Qiang Yang. Fedmkt: Federated mutual knowledge transfer for large and small language models. arXiv preprint arXiv:2406.02224, 2024

2024 arXiv
[88]

Federated learning with gan-based data synthesis for non-iid clients

Zijian Li, Jiawei Shao, Yuyi Mao, Jessie Hui Wang, and Jun Zhang. Federated learning with gan-based data synthesis for non-iid clients. In International Workshop on Trustworthy Federated Learning, pages 17–32. Springer, 2022

2022
[89]

Gfl: Federated learning on non-iid data via privacy-preserving synthetic data

Yihang Cheng, Lan Zhang, and Anran Li. Gfl: Federated learning on non-iid data via privacy-preserving synthetic data. In 2023 IEEE International Conference on Pervasive Computing and Communications (PerCom), pages 61–70. IEEE, 2023

2023
[90]

Harnessing large- language models to generate private synthetic text, 2024

Alexey Kurakin, Natalia Ponomareva, Umar Syed, Liam MacDermed, and Andreas Terzis. Harnessing large- language models to generate private synthetic text, 2024

2024
[91]

Privacy-preserving instructions for aligning large language models, 2024

Da Yu, Peter Kairouz, Sewoong Oh, and Zheng Xu. Privacy-preserving instructions for aligning large language models, 2024

2024
[92]

Stable federated learning with dataset condensation

Seong-Woong Kim and Dong-Wan Choi. Stable federated learning with dataset condensation. J. Comput. Sci. Eng., 16(1):52–62, 2022. 14 A PREPRINT - A PRIL 25, 2025

2022
[93]

Federated learning via decentralized dataset distillation in resource-constrained edge environments

Rui Song, Dai Liu, Dave Zhenyu Chen, Andreas Festag, Carsten Trinitis, Martin Schulz, and Alois Knoll. Federated learning via decentralized dataset distillation in resource-constrained edge environments. In 2023 International Joint Conference on Neural Networks (IJCNN), pages ...

2023
[94]

Distilled one-shot federated learning

Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning. arXiv preprint arXiv:2009.07999, 2020

2009 arXiv
[95]

Dataset distillation

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018

2018 arXiv
[96]

Dataset condensation with gradient matching

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05929, 2020

2006 arXiv
[97]

Dauphin, and David Lopez-Paz

Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018

2018
[98]

Mix2fld: Downlink federated learning after uplink federated distillation with two-way mixup

Seungeun Oh, Jihong Park, Eunjeong Jeong, Hyesung Kim, Mehdi Bennis, and Seong-Lyun Kim. Mix2fld: Downlink federated learning after uplink federated distillation with two-way mixup. IEEE Communications Letters, 24(10):2211–2215, 2020

2020
[99]

Differentially private synthetic data via foundation model APIs 1: Images

Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, Harsha Nori, and Sergey Yekhanin. Differentially private synthetic data via foundation model APIs 1: Images. In The Twelfth International Conference on Learning Representations, 2024

2024
[100]

Differentially Private Synthetic Data via Foundation Model APIs 2: Text

Chulin Xie, Zinan Lin, Arturs Backurs, Sivakanth Gopi, Da Yu, Huseyin A Inan, Harsha Nori, Haotian Jiang, Huishuai Zhang, Yin Tat Lee, et al. Differentially Private Synthetic Data via Foundation Model APIs 2: Text. In Forty-first International Conference on Machine Learning
[101]

Let’s synthesize step by step: Iterative dataset synthesis with large language models by extrapolating errors from small models

WANG Ruida, Wangchunshu Zhou, and Mrinmaya Sachan. Let’s synthesize step by step: Iterative dataset synthesis with large language models by extrapolating errors from small models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023

2023
[102]

Fedproto: Federated prototype learning across heterogeneous clients

Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8432–8440, 2022

2022
[103]

Federated learning from pre-trained models: A contrastive learning approach

Yue Tan, Guodong Long, Jie Ma, Lu Liu, Tianyi Zhou, and Jing Jiang. Federated learning from pre-trained models: A contrastive learning approach. Advances in Neural Information Processing Systems, 35:19332–19344, 2022

2022
[104]

FedPETuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models

Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. FedPETuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of th...

2023
[105]

Parameter-efficient tuning for large language model without calculating its gradients

Feihu Jin, Jiajun Zhang, and Chengqing Zong. Parameter-efficient tuning for large language model without calculating its gradients. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023

2023
[106]

Federated adaptation for foundation model-based recommendations

Chunxu Zhang, Guodong Long, Hongkuan Guo, Xiao Fang, Yang Song, Zhaojie Liu, Guorui Zhou, Zijian Zhang, Yang Liu, and Bo Yang. Federated adaptation for foundation model-based recommendations. arXiv preprint arXiv:2405.04840, 2024

2024 arXiv
[107]

Prefix-tuning: Optimizing continuous prompts for generation

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors,Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Jo...

2021
[108]

Fedprompt: Communication-efficient and privacy preserving prompt tuning in federated learning, 2023

Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Fedprompt: Communication-efficient and privacy preserving prompt tuning in federated learning, 2023

2023
[109]

Tunable soft prompts are messengers in federated learning

Chenhe Dong, Yuexiang Xie, Bolin Ding, Ying Shen, and Yaliang Li. Tunable soft prompts are messengers in federated learning. arXiv preprint arXiv:2311.06805, 2023

2023 arXiv
[110]

Efficient federated prompt tuning for black-box large pre-trained models, 2023

Zihao Lin, Yan Sun, Yifan Shi, Xueqian Wang, Lifu Huang, Li Shen, and Dacheng Tao. Efficient federated prompt tuning for black-box large pre-trained models, 2023

2023
[111]

Blackvip: Black-box visual prompting for robust transfer learning

Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTaek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, and Kyungwoo Song. Blackvip: Black-box visual prompting for robust transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p...

2023
[112]

Fate-llm: A industrial grade federated learning framework for large language models, 2023

Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. Fate-llm: A industrial grade federated learning framework for large language models, 2023. 15 A PREPRINT - A PRIL 25, 2025

2023
[113]

Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning, 2023

Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning, 2023

2023
[114]

Openfedllm: Training large language models on decentralized private data via federated learning, 2024

Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning, 2024

2024
[115]

Zoopfl: Exploring black-box foundation models for personalized federated learning, 2023

Wang Lu, Hao Yu, Jindong Wang, Damien Teney, Haohan Wang, Yiqiang Chen, Qiang Yang, Xing Xie, and Xiangyang Ji. Zoopfl: Exploring black-box foundation models for personalized federated learning, 2023

2023
[116]

Fedbiot: Llm local fine-tuning in federated learning without full model

Feijie Wu, Zitao Li, Yaliang Li, Bolin Ding, and Jing Gao. Fedbiot: Llm local fine-tuning in federated learning without full model. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, page 3345–3355, New York, NY , USA, 2024. Assoc...

2024
[117]

Terence Jie Chua, Wenhan Yu, Jun Zhao, and Kwok-Yan Lam. Fedpeat: Convergence of federated learning, parameter-efficient fine tuning, and emulator assisted tuning for artificial intelligence foundation models with mobile edge computing, 2024

2024
[118]

Knowledge Fusion of Large Language Models

Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge Fusion of Large Language Models. In Proceedings of The Twelfth International Conference on Learning Representations, 2024

2024
[119]

FuseChat: Knowledge Fusion of Chat Models

Fanqi Wan, Ziyi Yang, Longguang Zhong, Xiaojun Quan, Xinting Huang, and Wei Bi. FuseChat: Knowledge Fusion of Chat Models. arXiv preprint arXiv:2402.16107, 2024

2024 arXiv
[120]

Dataless Knowledge Fusion by Merging Weights of Language Models

Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless Knowledge Fusion by Merging Weights of Language Models. In The Eleventh International Conference on Learning Representations, 2023

2023
[121]

Composing parameter-efficient modules with arithmetic operation

Jinghan Zhang, Junteng Liu, Junxian He, et al. Composing parameter-efficient modules with arithmetic operation. Advances in Neural Information Processing Systems, 36:12589–12610, 2023

2023
[122]

Vertical federated learning: Concepts, advances, and challenges

Yang Liu, Yan Kang, Tianyuan Zou, Yanhong Pu, Yuanqin He, Xiaozhou Ye, Ye Ouyang, Ya-Qin Zhang, and Qiang Yang. Vertical federated learning: Concepts, advances, and challenges. IEEE Transactions on Knowledge and Data Engineering, 2024

2024
[123]

Split learning for health: Distributed deep learning without sharing raw patient data

Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint arXiv:1812.00564, 2018

2018 arXiv
[124]

Advances and open problems in federated learning

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and trends® in machine learning, 14(1–...

2021
[125]

Introducing edge intelligence to smart meters via federated split learning

Yehui Li, Dalin Qin, H Vincent Poor, and Yi Wang. Introducing edge intelligence to smart meters via federated split learning. Nature Communications, 15(1):9044, 2024

2024
[126]

Fedbert: When federated learning meets pre-training

Yuanyishu Tian, Yao Wan, Lingjuan Lyu, Dezhong Yao, Hai Jin, and Lichao Sun. Fedbert: When federated learning meets pre-training. ACM Trans. Intell. Syst. Technol., 13(4), August 2022

2022
[127]

Privatelora for efficient privacy preserving llm, 2023

Yiming Wang, Yu Lin, Xiaodong Zeng, and Guannan Zhang. Privatelora for efficient privacy preserving llm, 2023

2023
[128]

Sfprompt: Communication-efficient split federated fine-tuning for large pre-trained models over resource-limited devices, 2024

Linxiao Cao, Yifei Zhu, and Wei Gong. Sfprompt: Communication-efficient split federated fine-tuning for large pre-trained models over resource-limited devices, 2024

2024
[129]

A split-and-privatize framework for large language model fine-tuning

Xicong Shen, Yang Liu, Huiqi Liu, Jue Hong, Bing Duan, Zirui Huang, Yunlong Mao, Ye Wu, and Di Wu. A split-and-privatize framework for large language model fine-tuning. arXiv preprint arXiv:2312.15603, 2023

2023 arXiv
[130]

Fast and slow generating: An empirical study on large and small language models collaborative decoding

Kaiyan Zhang, Jianyu Wang, Ning Ding, Biqing Qi, Ermo Hua, Xingtai Lv, and Bowen Zhou. Fast and slow generating: An empirical study on large and small language models collaborative decoding. arXiv preprint arXiv:2406.12295, 2024

2024 arXiv
[131]

Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding

Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In Annual Meeting of the Association for Computational Linguistics, 2024

2024
[132]

Cascade speculative drafting for even faster llm inference

Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Jie Huang, and Kevin Chen-Chuan Chang. Cascade speculative drafting for even faster llm inference. ArXiv, abs/2312.11462, 2023

2023 arXiv
[133]

Fast inference from transformers via speculative decoding

Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR, 2023

2023
[134]

Speculative decoding with big little decoder

Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. Advances in Neural Information Processing Systems, 36, 2024. 16 A PREPRINT - A PRIL 25, 2025

2024
[135]

Spectr: Fast speculative decoding via optimal transport

Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36, 2024

2024
[136]

Online speculative decoding

Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Ion Stoica, Zhijie Deng, Alvin Cheung, and Hao Zhang. Online speculative decoding. arXiv preprint arXiv:2310.07177, 2023

2023 arXiv
[137]

Contrastive decoding: Open-ended text generation as optimization

Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61...

2023
[138]

Tuning language models by proxy

Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A Smith. Tuning language models by proxy. arXiv preprint arXiv:2401.08565, 2024

2024 arXiv
[139]

An emulator for fine-tuning large language models using small language models

Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D Manning. An emulator for fine-tuning large language models using small language models. arXiv preprint arXiv:2310.12962, 2023

2023 arXiv
[140]

On giant’s shoulders: Effortless weak to strong by dynamic logits fusion, 2024

Chenghao Fan, Zhenyi Lu, Wei Wei, Jie Tian, Xiaoye Qu, Dangyang Chen, and Yu Cheng. On giant’s shoulders: Effortless weak to strong by dynamic logits fusion, 2024

2024
[141]

Cpt: Consistent proxy tuning for black-box optimization, 2024

Yuanyang He, Zitong Huang, Xinxing Xu, Rick Siow Mong Goh, Salman Khan, Wangmeng Zuo, Yong Liu, and Chun-Mei Feng. Cpt: Consistent proxy tuning for black-box optimization, 2024

2024
[142]

CombLM: Adapting black-box language models through small fine-tuned models

Aitor Ormazabal, Mikel Artetxe, and Eneko Agirre. CombLM: Adapting black-box language models through small fine-tuned models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 29...

2023
[144]

Retrieval-augmented generation for large language models: A survey, 2024

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey, 2024

2024
[145]

Retrieval-augmented generation for knowledge-intensive nlp tasks

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In H. Larochelle, M. ...

2020
[146]

Improving the domain adaptation of retrieval augmented generation (RAG) models for open domain question answering

Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. Improving the domain adaptation of retrieval augmented generation (RAG) models for open domain question answering. Transactions of the Association for Computatio...

2023
[147]

REPLUG: Retrieval-augmented black-box language models

Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: Retrieval-augmented black-box language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North...

2024
[148]

Blade: Enhancing black-box large language models with small domain-specific models

Haitao Li, Qingyao Ai, Jia Chen, Qian Dong, Zhijing Wu, Yiqun Liu, Chong Chen, and Qi Tian. Blade: Enhancing black-box large language models with small domain-specific models. arXiv preprint arXiv:2403.18365, 2024

2024 arXiv
[150]

Training data extraction from pre-trained language models: A survey

Shotaro Ishihara. Training data extraction from pre-trained language models: A survey. arXiv preprint arXiv:2305.16157, 2023

2023 arXiv
[151]

Extracting training data from large language models

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-V oss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), page...

2021
[152]

Scalable extraction of training data from (production) language models

Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christo- pher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311...

2023 arXiv
[154]

H. Yin, A. Mallya, A. Vahdat, J. M. Alvarez, J. Kautz, and P. Molchanov. See through gradients: Image batch recovery via gradinversion. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16332–16341, Los Alamitos, CA, USA, jun 2021. IEEE Compu...

2021
[156]

Privacy risks of general-purpose language models

Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. Privacy risks of general-purpose language models. In 2020 IEEE Symposium on Security and Privacy (SP), pages 1314–1331. IEEE, 2020

2020
[157]

Information leakage in embedding models

Congzheng Song and Ananth Raghunathan. Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 377–390, 2020

2020
[158]

On the privacy risk of in-context learning

Haonan Duan, Adam Dziedzic, Mohammad Yaghini, Nicolas Papernot, and Franziska Boenisch. On the privacy risk of in-context learning. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023

2023
[159]

Calibrating noise to sensitivity in private data analysis

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Shai Halevi and Tal Rabin, editors, Theory of Cryptography, pages 265–284, Berlin, Heidelberg,
[160]

Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang

Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS ’16, page 308–318, New York, NY , USA, 2...

2016
[161]

Extracting training data from large language models

Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-V oss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Symposium (U...

2021
[162]

CAPE: Context-aware private embeddings for private language learning

Richard Plant, Dimitra Gkatzia, and Valerio Giuffrida. CAPE: Context-aware private embeddings for private language learning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natura...

2021
[163]

Differentially private representation for NLP: Formal guarantee and an empirical study on privacy and fairness

Lingjuan Lyu, Xuanli He, and Yitong Li. Differentially private representation for NLP: Formal guarantee and an empirical study on privacy and fairness. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages...

2020
[164]

Towards differentially private text representations

Lingjuan Lyu, Yitong Li, Xuanli He, and Tong Xiao. Towards differentially private text representations. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1813–1816, 2020

2020
[165]

Natural language understanding with privacy-preserving bert

Chen Qu, Weize Kong, Liu Yang, Mingyang Zhang, Michael Bendersky, and Marc Najork. Natural language understanding with privacy-preserving bert. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, CIKM ’21, page 1488–1497, New York, NY...

2021
[166]

A customized text sanitization mechanism with differential privacy

Sai Chen, Fengran Mo, Yanhao Wang, Cen Chen, Jian-Yun Nie, Chengyu Wang, and Jamie Cui. A customized text sanitization mechanism with differential privacy. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistic...

2023
[167]

Privacy-preserving prompt tuning for large language model services, 2023

Yansong Li, Zhixing Tan, and Yang Liu. Privacy-preserving prompt tuning for large language model services, 2023

2023
[168]

Flocks of stochastic parrots: differentially private prompt learning for large language models

Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. Flocks of stochastic parrots: differentially private prompt learning for large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hoo...

2024
[169]

Wang, Chenhui Zhang, Zhangheng LI, Bo Li, and Zhangyang Wang

Junyuan Hong, Jiachen T. Wang, Chenhui Zhang, Zhangheng LI, Bo Li, and Zhangyang Wang. DP-OPT: Make large language model your privacy-preserving prompt engineer. In The Twelfth International Conference on Learning Representations, 2024

2024
[170]

Remoterag: A privacy-preserving llm cloud rag service, 2024

Yihang Cheng, Lan Zhang, Junyang Wang, Mu Yuan, and Yunhao Yao. Remoterag: A privacy-preserving llm cloud rag service, 2024

2024
[171]

Privacy-preserving retrieval augmented generation with differential privacy, 2024

Tatsuki Koga, Ruihan Wu, and Kamalika Chaudhuri. Privacy-preserving retrieval augmented generation with differential privacy, 2024

2024
[172]

Towards robust and privacy-preserving text representations

Yitong Li, Timothy Baldwin, and Trevor Cohn. Towards robust and privacy-preserving text representations. In Iryna Gurevych and Yusuke Miyao, editors,Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 25–30, M...

2018
[173]

Privacy-preserving neural representations of text

Maximin Coavoux, Shashi Narayan, and Shay B Cohen. Privacy-preserving neural representations of text. arXiv preprint arXiv:1808.09408, 2018

2018 arXiv
[174]

Efficient adversarial training in llms with continuous attacks, 2024

Sophie Xhonneux, Alessandro Sordoni, Stephan Günnemann, Gauthier Gidel, and Leo Schwinn. Efficient adversarial training in llms with continuous attacks, 2024

2024
[175]

Roundy, and Jasjeet Dhaliwal

Saurabh Shintre, Kevin A. Roundy, and Jasjeet Dhaliwal. Making machine learning forget. In Annual Privacy Forum, 2019

2019
[176]

Knowledge unlearning for mitigating privacy risks in language models

Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Proceedings of the 61st Annual Meeting o...

2023
[177]

Offset unlearning for large language models

James Y Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen. Offset unlearning for large language models. arXiv preprint arXiv:2404.11045, 2024

2024 arXiv
[178]

Privacy preserving prompt engineering: A survey

Kennedy Edemacu and Xintao Wu. Privacy preserving prompt engineering: A survey. arXiv preprint arXiv:2404.06001, 2024

2024 arXiv
[179]

The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization

Ildikó Pilán, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David Sánchez, and Montserrat Batet. The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization. Computational Linguistics, 48(4):1053–1101, 12 2022

2022
[180]

Hide and seek (has): A lightweight framework for prompt privacy protection

Yu Chen, Tingxin Li, Huiming Liu, and Yang Yu. Hide and seek (has): A lightweight framework for prompt privacy protection. arXiv preprint arXiv:2309.03057, 2023

2023 arXiv
[181]

Protecting user privacy in remote conversational systems: A privacy-preserving framework based on text sanitization

Zhigang Kan, Linbo Qiao, Hao Yu, Liwen Peng, Yifu Gao, and Dongsheng Li. Protecting user privacy in remote conversational systems: A privacy-preserving framework based on text sanitization. arXiv preprint arXiv:2306.08223, 2023

2023 arXiv
[182]

Casper: Prompt sanitization for protecting user privacy in web-based large language models

Chun Jie Chong, Chenxi Hou, Zhihao Yao, and Seyed Mohammadjavad Seyed Talebi. Casper: Prompt sanitization for protecting user privacy in web-based large language models. arXiv preprint arXiv:2408.07004, 2024

2024
[183]

TextFusion: Privacy-preserving pre-trained model inference via token fusion

Xin Zhou, Jinzhu Lu, Tao Gui, Ruotian Ma, Zichu Fei, Yuran Wang, Yong Ding, Yibo Cheung, Qi Zhang, and Xuanjing Huang. TextFusion: Privacy-preserving pre-trained model inference via token fusion. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2...

2022
[184]

Textobfuscator: Making pre-trained language model a privacy protector via obfuscating word representations

Xin Zhou, Yi Lu, Ruotian Ma, Tao Gui, Yuran Wang, Yong Ding, Yibo Zhang, Qi Zhang, and Xuan-Jing Huang. Textobfuscator: Making pre-trained language model a privacy protector via obfuscating word representations. In Findings of the Association for Computational Linguistics: ACL...

2023
[185]

Privacy-preserving language model inference with instance obfuscation, 2024

Yixiang Yao, Fei Wang, Srivatsan Ravi, and Muhao Chen. Privacy-preserving language model inference with instance obfuscation, 2024

2024
[186]

Latticegen: A cooperative framework which hides generated text in a lattice for privacy-aware generation on cloud

Mengke Zhang, Tianxing He, Tianle Wang, Lu Mi, Fatemehsadat Mireshghallah, Binyi Chen, Hao Wang, and Yulia Tsvetkov. Latticegen: A cooperative framework which hides generated text in a lattice for privacy-aware generation on cloud. arXiv preprint arXiv:2309.17157, 2023

2023 arXiv
[187]

Fedml-he: An efficient homomorphic-encryption-based privacy-preserving federated learning system, 2024

Weizhao Jin, Yuhang Yao, Shanshan Han, Jiajun Gu, Carlee Joe-Wong, Srivatsan Ravi, Salman Avestimehr, and Chaoyang He. Fedml-he: An efficient homomorphic-encryption-based privacy-preserving federated learning system, 2024. 19 A PREPRINT - A PRIL 25, 2025

2024
[188]

THE-X: Privacy-preserving transformer inference with homomorphic encryption

Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. THE-X: Privacy-preserving transformer inference with homomorphic encryption. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the ...

2022
[189]

Puma: Secure inference of llama-7b in five minutes, 2023

Ye Dong, Wen jie Lu, Yancheng Zheng, Haoqi Wu, Derun Zhao, Jin Tan, Zhicong Huang, Cheng Hong, Tao Wei, and Wenguang Chen. Puma: Secure inference of llama-7b in five minutes, 2023

2023
[190]

Ob- fuscatune: Obfuscated offsite fine-tuning and inference of proprietary llms on private datasets

Ahmed Frikha, Nassim Walha, Ricardo Mendes, Krishna Kanth Nakka, Xue Jiang, and Xuebing Zhou. Ob- fuscatune: Obfuscated offsite fine-tuning and inference of proprietary llms on private datasets. arXiv preprint arXiv:2407.02960, 2024

2024 arXiv
[191]

Mea-defender: A robust watermark against model extraction attack

Peizhuo Lv, Hualong Ma, Kai Chen, Jiachen Zhou, Shengzhi Zhang, Ruigang Liang, Shenchen Zhu, Pan Li, and Yingjun Zhang. Mea-defender: A robust watermark against model extraction attack. In 2024 IEEE Symposium on Security and Privacy (SP), pages 2515–2533, 2024

2024
[192]

Model leeching: An extraction attack targeting llms, 2023

Lewis Birch, William Hackett, Stefan Trawicki, Neeraj Suri, and Peter Garraghan. Model leeching: An extraction attack targeting llms, 2023

2023
[193]

I know what you trained last summer: A survey on stealing machine learning models and defences

Daryna Oliynyk, Rudolf Mayer, and Andreas Rauber. I know what you trained last summer: A survey on stealing machine learning models and defences. ACM Computing Surveys, 55(14s):1–41, 2023

2023
[194]

Watermarking llms with weight quantization, 2023

Linyang Li, Botian Jiang, Pengyu Wang, Ke Ren, Hang Yan, and Xipeng Qiu. Watermarking llms with weight quantization, 2023

2023
[195]

A watermark for large language models, 2024

John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models, 2024

2024
[196]

Instructional fingerprinting of large language models, 2024

Jiashu Xu, Fei Wang, Mingyu Derek Ma, Pang Wei Koh, Chaowei Xiao, and Muhao Chen. Instructional fingerprinting of large language models, 2024

2024
[197]

Hide and seek: Fingerprinting large language models with evolutionary learning, 2024

Dmitri Iourovitski, Sanat Sharma, and Rakshak Talwar. Hide and seek: Fingerprinting large language models with evolutionary learning, 2024

2024
[198]

Taylor unswift: Secured weight release for large language models via taylor expansion

Guanchu Wang, Yu-Neng Chuang, Ruixiang Tang, Shaochen Zhong, Jiayi Yuan, Hongye Jin, Zirui Liu, Vipin Chaudhary, Shuai Xu, James Caverlee, et al. Taylor unswift: Secured weight release for large language models via taylor expansion. arXiv preprint arXiv:2410.05331, 2024

arXiv
[199]

Lora: Low-rank adaptation of large language models

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

2021 arXiv
[200]

P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks

Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting o...

2022
[201]

Federated fine-tuning of llms on the very edge: The good, the bad, the ugly

Herbert Woisetschläger, Alexander Isenko, Shiqiang Wang, Ruben Mayer, and Hans-Arno Jacobsen. Federated fine-tuning of llms on the very edge: The good, the bad, the ugly. arXiv preprint arXiv:2310.03150, 2023

2023 arXiv
[202]

Lee, Bugra Akyildiz, Maximilian Balandat, Joe Spisak, Ravi Jain, Mike Rabbat, and Kim Hazelwood

Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga Behram, James Huang, Charles Bai, Michael Gschwind, Anurag Gupta, Myle Ott, Anastasia Melnikov, Salvatore Candido, David Brooks, Geeta Chauhan, Benjamin Lee, Hsien-...

2022
[203]

Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V . S. Laksh- manan, and Ahmed Hassan Awadallah. Hybrid llm: Cost-efficient and quality-aware query routing, 2024

2024
[204]

Frugalgpt: How to use large language models while reducing cost and improving performance, 2023

Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance, 2023

2023
[205]

Privacy for free: How does dataset condensation help privacy? In International Conference on Machine Learning, pages 5378–5396

Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? In International Conference on Machine Learning, pages 5378–5396. PMLR, 2022

2022
[206]

Maas, Raymond E

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y . Ng, and Christopher Potts. Learning Word Vectors for Sentiment Analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150...

2011
[207]

Position: Considerations for differentially private learning with large-scale public pretraining

Florian Tramèr, Gautam Kamath, and Nicholas Carlini. Position: Considerations for differentially private learning with large-scale public pretraining. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria 20 A PREPRINT - A PRIL 25, 2025 Oliver, Jonathan ...

2025
[208]

Trading off privacy, utility, and efficiency in federated learning

Xiaojin Zhang, Yan Kang, Kai Chen, Lixin Fan, and Qiang Yang. Trading off privacy, utility, and efficiency in federated learning. ACM Transactions on Intelligent Systems and Technology, 14(6):1–32, 2023

2023
[209]

Privacy, accuracy, and model fairness trade-offs in federated learning

Xiuting Gu, Zhu Tianqing, Jie Li, Tao Zhang, Wei Ren, and Kim-Kwang Raymond Choo. Privacy, accuracy, and model fairness trade-offs in federated learning. Computers & Security, 122:102907, 2022

2022
[210]

Breaking the communication-privacy-accuracy trilemma

Wei-Ning Chen, Peter Kairouz, and Ayfer Ozgur. Breaking the communication-privacy-accuracy trilemma. Advances in Neural Information Processing Systems, 33:3312–3324, 2020

2020
[211]

You only look once: Unified, real-time object detection

J Redmon. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016

2016
[212]

Yolo9000: better, faster, stronger

Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7263–7271, 2017

2017
[213]

Yolov3: An incremental improvement

Joseph Redmon. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018

2018 arXiv
[214]

Yolov4: Optimal speed and accuracy of object detection

Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934, 2020

2004 arXiv
[215]

Yolov6: A single-stage object detection framework for industrial applications

Chuyi Li, Lulu Li, Hongliang Jiang, Kaiheng Weng, Yifei Geng, Liang Li, Zaidan Ke, Qingyuan Li, Meng Cheng, Weiqiang Nie, et al. Yolov6: A single-stage object detection framework for industrial applications. arXiv preprint arXiv:2209.02976, 2022

2022 arXiv
[216]

Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors

Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7464–7475, 2023

2023
[217]

High-resolution image synthesis with latent diffusion models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

2022
[218]

On-device language models: A comprehensive review, 2024

Jiajun Xu, Zhiyuan Li, Wei Chen, Qun Wang, Xin Gao, Qi Cai, and Ziyuan Ling. On-device language models: A comprehensive review, 2024

2024
[219]

Towards federated learning at scale: Syste m design

Keith Bonawitz. Towards federated learning at scale: Syste m design. arXiv preprint arXiv:1902.01046, 2019. 21

1902 arXiv
[2006]

Springer Berlin Heidelberg
[2023]

Association for Computational Linguistics

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [155]

Breaching fedmd: Image recovery via paired-logits inversion attack

Hideaki Takahashi, Jingjing Liu, and Yang Liu. Breaching fedmd: Image recovery via paired-logits inversion attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12198–12207, 2023

2023

[2] [153]

Deep leakage from gradients

Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. dÁlché-Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Systems 32, pages 14774–14784. Curran Associates, Inc., 2019

2019

[3] [149]

The good and the bad: Exploring privacy issues in retrieval-augmented generation (RAG)

Shenglai Zeng, Jiankun Zhang, Pengfei He, Yiding Liu, Yue Xing, Han Xu, Jie Ren, Yi Chang, Shuaiqiang Wang, Dawei Yin, and Jiliang Tang. The good and the bad: Exploring privacy issues in retrieval-augmented generation (RAG). In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, page...

2024

[4] [143]

CoGenesis: A framework collaborating large and small language models for secure context-aware instruction following

Kaiyan Zhang, Jianyu Wang, Ermo Hua, Biqing Qi, Ning Ding, and Bowen Zhou. CoGenesis: A framework collaborating large and small language models for secure context-aware instruction following. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

2024

[5] [1]

Will we run out of data? limits of llm scaling based on human-generated data, 2024

Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Will we run out of data? limits of llm scaling based on human-generated data, 2024

2024

[6] [2]

Advances and open challenges in federated foundation models, 2024

Chao Ren, Han Yu, Hongyi Peng, Xiaoli Tang, Bo Zhao, Liping Yi, Alysa Ziying Tan, Yulan Gao, Anran Li, Xiaoxiao Li, Zengxiang Li, and Qiang Yang. Advances and open challenges in federated foundation models, 2024

2024

[7] [3]

Shenghui Li, Fanghua Ye, Meng Fang, Jiaxu Zhao, Yun-Hin Chan, Edith C. H. Ngai, and Thiemo V oigt. Synergizing foundation models and federated learning: A survey, 2024

2024

[8] [4]

Introducing chatgpt

OpenAI. Introducing chatgpt. OpenAI, 2022

2022