{"id":"2927dd37-2bdb-4efd-ac22-92a8a122c69d","arxiv_id":"2504.17421","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper organizes large-small model collaboration into downward, upward, and inference-time transfer, and advocates multi-objective benchmarks for private-domain tasks.","lead":"This preprint is a survey and position paper on letting large language models and small domain-specific models collaborate across organizational boundaries under privacy and resource limits. It proposes a taxonomy of collaboration directions and argues for multi-objective, application-driven benchmarks on real private data.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central advocacy rests on an unverified operating point: no surveyed cross-boundary carrier is shown to transfer enough domain knowledge while satisfying Eq. (1)'s privacy, security, and resource constraints, and §4.3 concedes the metrics needed to check this do not yet exist.","rationale":"The reader's weakest assumption is exactly the enabling condition I identify: knowledge carriers must transfer useful information while satisfying cross-boundary constraints. The paper is explicitly a position paper, so absence of experiments alone is not disqualifying; the load-bearing issue is that the central advocacy presupposes a feasible operating point for cross-boundary transfer, and the paper itself lists the missing pieces: no standardized privacy metrics, no real-world private benchmarks (§4.3), and known attacks against logits/representations and intermediate outputs (§4.1). This is an internal gap between the advocacy and the evidence, not a disagreement with external consensus. A single controlled experiment instantiating Eq. (1)/(2) with concrete metrics would determine whether the premise holds. Since this is the same concern the reader already used to justify CONDITIONAL, the verdict is unchanged.","tokens_in":24691,"tokens_out":7617,"duration_ms":75392,"concrete_test":"Run one representative downward method (offsite-tuning [49]) and one upward method (CombLM [142]) on a single private-domain task with a fixed privacy metric (membership-inference AUC or DP-ε), a fixed per-round communication budget (as discussed in §4.2), and a fixed local compute budget. Check whether any configuration reaches the local-SM baseline utility while satisfying Eq. (1)'s constraints. If no configuration does, the collaborative operating point the paper advocates is not demonstrated; if one does, the concern is settled in the paper's favor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that LM–SM collaboration can accelerate private-domain adaptation—holds only if the information carriers surveyed in §3.4.1–3.4.4 (logits/representations, synthetic data, adapters/prompts) can transfer enough task-relevant signal across the silo boundary while keeping privacy leakage M_p, model leakage M_L, and resource cost M_e within the bounds of Eq. (1). The paper never demonstrates this. It cites concrete privacy breaks against the carriers it advocates (paired-logits inversion in Breaching FedMD [155]; gradient inversion [153]; RAG privacy risks [149]), and §4.3 concedes that 'quantifying the privacy implications of emerging privacy-preserving techniques using DP remains an open challenge' and that real-world private benchmarks are absent. Because M_p/M_L/M_e are never instantiated, Eq. (1) remains a template rather than a checkable feasibility problem, and Eq. (2)'s 'flexible' f can rank methods arbitrarily. The enabling condition is thus asserted rather than established, and the paper's own limitation statements confirm the gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This position paper argues that collaboration between large language models (LMs) and small models (SMs) can accelerate domain adaptation in private, cross-boundary settings where data, model parameters, and resources are constrained. The paper formalizes the collaboration as a constrained optimization problem in Eq. (1), proposes a taxonomy of three research directions (knowledge transfer from LMs to SMs, from SMs to LMs, and inference-time collaboration), and surveys representative techniques in each direction. It then discusses privacy threats, efficiency considerations, and benchmarking gaps, and proposes a flexible multi-objective evaluation framework in Eq. (2). The paper closes with a call for application-driven research and real-world private benchmarks, including an anecdotal example of a deployed industrial system.","tokens_in":24984,"tokens_out":4516,"duration_ms":44194,"significance":"If the central thesis is accepted, the paper provides a useful organizing framework for a rapidly growing and fragmented literature, connecting model collaboration to privacy, security, and efficiency constraints in a way that many prior surveys do not. The problem formulation in Eq. (1) and the proposed taxonomy could serve as a shared vocabulary for future work. The paper also usefully highlights the lack of standardized privacy metrics and real-world private benchmarks, which is an important gap. However, the paper makes no quantitative claims and does not establish its central enabling condition: that the surveyed knowledge carriers (logits, representations, synthetic data, adapters, prompts) can transfer sufficient task-relevant information while satisfying the constraints in Eq. (1). The proposed benchmark in Eq. (2) is not yet operational, and the survey's \"comprehensive\" coverage is not supported by a systematic methodology. These issues limit the paper's current contribution to a roadmap rather than a validated framework.","major_comments":[{"comment":"The paper's central advocacy, stated in the abstract and Section 1, is that LM–SM collaboration can accelerate private-domain adaptation. This claim depends on the premise that the information carriers surveyed in §3.4.1–3.4.4 can transfer enough task-relevant signal across the silo boundary while keeping privacy leakage, model leakage, and resource costs within the bounds of Eq. (1). The paper never demonstrates this premise. Section 4.1 cites concrete privacy breaks against these very carriers (e.g., paired-logits inversion for FedMD in [155], gradient inversion in [153], RAG privacy risks in [149]), and Section 4.3 concedes that \"quantifying the privacy implications of emerging privacy-preserving techniques using DP remains an open challenge\" and that real-world private benchmarks are absent. Eq. (1) therefore remains a template rather than a checkable feasibility problem. The authors should either synthesize the available positive and negative empirical evidence on whether these carriers can operate within the constraints, or explicitly reframe the central claim as a hypothesis that motivates a research agenda.","section":"§4.3 and §4.1"},{"comment":"The proposed multi-objective benchmark defines a scoring function f that is \"flexible\" and can be \"carefully designed,\" but no concrete instantiation is given. As written, f can be chosen to rank any method arbitrarily, so the benchmark is not falsifiable. To be usable, the paper should propose at least one default f (e.g., a weighted linear combination), specify a protocol for reporting the individual objectives o_1, ..., o_n, and state how trade-offs are to be handled without post hoc reweighting.","section":"§5.1, Eq. (2)"},{"comment":"The paper claims that its taxonomy classifies the literature into three complete directions, but no systematic methodology—such as a search protocol, inclusion/exclusion criteria, or coverage statistics—is provided to support the \"comprehensive\" claim. In addition, the taxonomy contains internal inconsistencies: Table 1 lists collaborative decoding as transferring \"insensitive data\" from S to L, whereas the text in §3.4.4 states that \"these approaches directly expose private data to LMs during inference time,\" with CoGenesis [143] being the exception. This inconsistency undermines the reliability of the taxonomy as a classification of the field.","section":"§3.4, Table 1"},{"comment":"The paper cites as evidence a deployed AsiaInfo system for wireless network complaints, claiming that \"test results show that it not only significantly enhanced user experience but also greatly improved the efficiency of conducting data analysis.\" No data, evaluation protocol, or metrics are provided. If this deployment is intended to support the central thesis, the paper should either include concrete evaluation results or clearly mark the passage as an anecdotal illustration rather than evidence.","section":"§5.2"}],"minor_comments":[{"comment":"There is a missing space in \"utilizeSmall Models\" in the third paragraph of Section 2.","section":"§2"},{"comment":"References [1] and [19] appear to cite the same work (Villalobos et al., \"Will we run out of data?\") in two versions; this should be consolidated or cross-referenced.","section":"References"},{"comment":"Reference [219] contains a typo: \"Syste m design\" should be \"System design.\"","section":"References"},{"comment":"The phrase \"Table 1 and Figure 2 summarized\" should be \"Table 1 and Figure 2 summarize\" for subject-verb agreement.","section":"§3.4"}],"recommendation":"major_revision","confidential_remarks":"The survey repeatedly anchors its categories on the authors' own prior works (e.g., FedTGP [59], FuseGen [74], CrossLM [77], and the split-and-privatize framework [129]). This is not inappropriate per se, but combined with the absence of a systematic selection methodology, it creates a risk that the \"comprehensive\" framing will be perceived as reflecting the authors' research program rather than the field at large. The editor may wish to request that the authors add a broader set of independent anchor examples and disclose the survey methodology. The paper is positioned as a survey/position paper; if the journal's scope emphasizes validated technical contributions over roadmaps, this should be weighed in the decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work in federated or privacy-preserving LLM adaptation. The real contribution is organizational: the paper groups the scattered literature on large-small model collaboration into three directions—downward transfer, upward transfer, and inference-time collaboration—and ties them to a constrained optimization view (Eq. 1) that makes the privacy/security/efficiency trade-offs explicit. That framing is useful and mostly faithful to the cited work. I don't see derivation errors because there are no derivations; the equations are definitions, and the paper doesn't claim otherwise.\n\nWhat it does well: the taxonomy is sensible, the coverage is broad, and Section 4 does a decent job cataloging defenses (DP, adversarial training, unlearning, sanitization, secure computation) and efficiency considerations. The paper is also honest about the field's gaps: Section 4.3 concedes that privacy metrics are not standardized and real-world private benchmarks are missing. That honesty is a point in its favor.\n\nWhere it wobbles: the \"comprehensive overview\" claim in the abstract oversells what is essentially a curated, narrative survey—there is no systematic search or inclusion protocol, so coverage is likely biased toward the authors' own network. The paper also leans on the authors' own methods (FuseGen, FedTGP, CrossLM, split-and-privatize) when motivating categories; that's not disqualifying, but a reader should be aware. The bigger soft spot is the load-bearing advocacy: the claim that LM-SM collaboration can accelerate private-domain adaptation is asserted, not demonstrated. The paper cites real privacy breaks (paired-logits inversion, gradient inversion, RAG leaks) against the very carriers it advocates, then concedes we don't yet have metrics to check whether the trade-off works. That's a fair state of the field, but it means the paper is a research agenda, not a validated approach. The one anecdotal deployment mention (AsiaInfo) is too thin to count as evidence.\n\nWho it's for: newcomers to federated or privacy-preserving LLM adaptation who want a map. Researchers already in the area won't find much new. With revisions to tone down the comprehensiveness claim and separate the taxonomy from the advocacy, it would be a solid survey. As is, it deserves a serious referee but not unconditional acceptance.","headline":"A serviceable survey/position paper with a sensible taxonomy; the central advocacy is a research agenda rather than a demonstrated result, and the paper is honest about that.","tokens_in":25473,"tokens_out":2212,"would_cite":false,"duration_ms":20920,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that private-domain AI should be built by pairing large general-purpose models with small on-site models, moving only distilled knowledge across the boundary instead of raw data or full model weights.","keywords":["large language models","small models","knowledge distillation","federated learning","data privacy","private-domain adaptation","cross-silo collaboration","multi-objective benchmarking"],"falsifier":"Run the best cross-silo collaboration on a genuinely private real-world dataset against three baselines—training the small model locally, fine-tuning the large model centrally on the same data, and standard federated learning—and measure utility, privacy leakage (for example, success of reconstruction or membership attacks), communication, computation, and query cost. If the collaborative method is not Pareto-better than at least one baseline on the multi-objective comparison, the paper's central advocacy is defeated. The paper itself notes the lack of standardized privacy metrics and real private benchmarks, so designing this comparison is the direct test of its thesis.","tokens_in":24529,"feed_emoji":"🤝","tokens_out":7777,"duration_ms":70362,"temperature":0.7,"pith_summary":"The paper argues that private-domain AI should be built by pairing a large general-purpose model with small on-site models rather than choosing between them. Its thesis is that the large model's breadth and the small model's domain specialization can be exchanged as knowledge carriers—logits and representations, synthetic data, or adapters and prompts—without moving raw data or exposing proprietary weights. The authors survey the literature through three directions: downward transfer from large to small models, upward transfer from small to large models, and inference-time collaboration, and they unify these under one constrained optimization problem with privacy, model-leakage, and resource budgets. They also propose a multi-objective benchmark so that methods can be compared on utility, privacy, efficiency, and query cost together rather than accuracy alone. The paper is explicit in Section 4.3 that the field lacks standardized privacy metrics and real-world private benchmarks, so the contribution is a framing and a research agenda rather than an evaluated method.","feed_headline":"Small models can ferry private data to large ones safely","feed_subtitle":"Three knowledge-transfer routes and one multi-objective benchmark for judging private-domain collaboration.","key_machinery":"The load-bearing idea is the knowledge carrier, the compact artifact that crosses the boundary between the data owner and the model owner in place of raw data and full weights. The paper identifies three families: logits and representations, synthetic data, and adapters or prompts; each can travel downward from large to small models, upward from small to large models, or both at inference time. Carriers do the work of separating the utility objective from the privacy, model-security, and resource constraints, so the tractability of the whole paradigm depends on whether these artifacts preserve enough task-relevant information to justify the collaboration.","core_discovery":"The central claim is that large and small models are complements: the large model contributes general knowledge, generative ability, and reasoning; the small model contributes private-domain specialization, on-site efficiency, and data access that the large model is forbidden to see. Knowledge can flow in both directions—distilled logits or representations, synthetic datasets, and lightweight adapters or prompts are the carriers—so a domain task can be solved without centralizing data and without releasing the large model's parameters. The paper formalizes this as a constrained optimization: maximize task utility while keeping privacy leakage, model leakage, and local computation within prescribed budgets. Its taxonomy claims that every existing approach fits one of three directions: downward transfer, upward transfer, or inference-time collaboration, with split learning, collaborative decoding, and retrieval as the inference-time family. The authors position this as the pattern for adapting large models to private domains and call for application-driven, multi-objective evaluation on real private data as the way to test it.","pith_inferences":["A natural experiment the paper does not run: hold one large model fixed and compare the three carriers—logits, synthetic data, and adapters—on the same private task under the same privacy budget; the result would show which carrier actually deserves investment.","If strong privacy guarantees force the carriers to be heavily sanitized, the argument implicitly caps the paradigm's ceiling at the small model's domain coverage, a consequence the authors do not spell out.","The proposed multi-objective benchmark could be instantiated as a Pareto frontier over utility, privacy leakage, communication cost, and query cost; leaving the scoring function unspecified is reasonable for a position paper, but a concrete instantiation would make the framework testable.","The paper mentions one industrial deployment as evidence of feasibility; a public case study reporting utility, privacy, and cost numbers on that system would convert the position into evidence."],"forward_implications":["Domain adaptation would no longer require shipping raw private data or full model weights; knowledge could travel as distilled predictions, synthetic samples, prototypes, or trained adapters.","Comparisons of collaboration strategies should be framed by the constrained problem with privacy, model-leakage, and resource budgets rather than by accuracy alone.","Benchmarks on open datasets such as IMDB overstate real performance because the domain gap to true private data is minimal, so private datasets that report the domain gap are needed.","A customizable multi-objective scoring function can rank methods on utility, privacy, communication, computation, and large-model query cost at once.","Inference-time collaboration extends the same privacy-respecting knowledge flow to settings where fine-tuning either model is infeasible, using split learning, collaborative decoding, or retrieval."],"supporting_citations":[{"why":"Supplies the distillation mechanism of teacher logits to student that underlies most large-to-small transfer.","marker":"[35]"},{"why":"Defines federated learning, the multi-party setting that motivates and carries small-to-large knowledge transfer.","marker":"[21]"},{"why":"Introduces offsite tuning, a parameter-based route that transfers a compressed emulator instead of the full model.","marker":"[49]"},{"why":"Demonstrates mutual enhancement of a large and small model through cross-silo generation-based knowledge transfer.","marker":"[77]"},{"why":"Shows small models guiding large-model generation and fusing large-model knowledge, grounding the upward generation-based route.","marker":"[74]"},{"why":"Establishes split or vertical federated learning, the basis for inference-time collaboration with intermediate outputs.","marker":"[122]"},{"why":"Blends a small fine-tuned model's distribution with a large black-box model's, grounding collaborative decoding.","marker":"[142]"}],"fun_headline_variants":["Big and small models team up for private-domain AI","Two-model collaboration: privacy, security, and efficiency","Downward and upward: knowledge transfer in model pairs","Hybrid models: meet privacy needs without centralizing data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The enabling premise is that the small carriers—logits, synthetic samples, prototypes, adapters, and prompts—can carry enough task-relevant knowledge across a privacy and security boundary while genuinely satisfying privacy, model-IP, and resource constraints; the paper surveys techniques but does not demonstrate this.","fun_headline_variants_meta":{"raw":{"variants":["Big and small models team up for private-domain AI","Two-model collaboration: privacy, security, and efficiency","Downward and upward: knowledge transfer in model pairs","Hybrid models: meet privacy needs without centralizing data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000299,"raw_usage":{"total_tokens":1758,"prompt_tokens":1002,"completion_tokens":756,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":692}},"tokens_in":618,"tokens_out":756,"duration_ms":7014,"temperature":1.0,"reasoning_tokens":692,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:40:05.365981+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the best cross-silo collaboration on a genuinely private real-world dataset against three baselines—training the small model locally, fine-tuning the large model centrally on the same data, and standard federated learning—and measure utility, privacy leakage (for example, success of reconstruction or membership attacks), communication, computation, and query cost. If the collaborative method is not Pareto-better than at least one baseline on the multi-objective comparison, the paper's central advocacy is defeated. The paper itself notes the lack of standardized privacy metrics and real private benchmarks, so designing this comparison is the direct test of its thesis.","supporting_citations":[{"cited_title":"FuseGen: PLM Fusion for Data-generation based Zero-shot Learning","cited_arxiv_id":"2406.12527","evidence_quote":"Shows small models guiding large-model generation and fusing large-model knowledge, grounding the upward generation-based route."}],"review_version":1}