{"id":"00213aa0-91a1-4c71-8694-f37c63e3a7be","arxiv_id":"2502.01804","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Soup-of-Experts pretrains a shared parameter bank and many expert vectors, plus a router, so a small specialist language model can be instantiated instantly from any domain-weight mixture without retraining.","lead":"This paper introduces Soup-of-Experts, a pretraining scheme where a large bank of expert parameters plus a small router can instantly produce a small specialist language model for any desired data mix. It matters because teams that need to ship many domain-specific small models could replace many retraining runs with one pretraining run and a cheap instantiation step.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline comparison is confounded: CRISP and domain experts ran 128K iterations (17B tokens) per model versus 1024K iterations (134B) for Soup-of-Experts, so 'best specialists' may reflect training budget rather than method.","rationale":"The reader's explicit weakest_assumption is the h_spe representation, but the reader's rationale already identifies the compute-budget confound as a primary reason for the CONDITIONAL verdict. I agree with that conditional posture, but I regard the unequal per-model training budget as the single most load-bearing concern because it directly threatens the empirical headline that Soup-of-Experts outperforms all baselines on specialist domains. The h_spe concern is real but secondary: it affects the generalization of the full specialization pipeline to arbitrary target distributions, whereas the compute-budget issue undermines the reported comparison even for the 16 Pile domains actually tested. A direct iso-token CRISP rerun is a concrete, decisive check. In good faith, the paper does have independent support: held-out Pile evaluation, several ablations, and fine-tuning results consistent with the method's claims. The central architecture and training objective are plausible; the concern is specifically that the evidence for 'best performance' is not yet clean.","tokens_in":16207,"tokens_out":7528,"duration_ms":78455,"concrete_test":"Retrain CRISP on each of the 16 Pile h_spe mixtures for 1024K iterations (134B tokens), matching the Soup-of-Experts pretraining budget per specialist, and compare per-domain and average specialized losses against Fig. 4 (at minimum, do this on 2-3 representative domains such as arxiv, uspto, and ubuntu). If fully trained CRISP reaches or beats the Soup-of-Experts specialized losses, the headline advantage is an artifact of the 8x budget gap; if Soup-of-Experts still wins at iso-tokens, the central comparison is strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the unequal training budget in the headline comparison (Fig. 4). Appendix A states that for the main experiments, Soup-of-Experts and generic pretraining run 1024K iterations (134B tokens), while domain experts and CRISP run only 128K iterations (17B tokens). The x-axis of Fig. 4 is total training time, so showing CRISP and domain experts below Soup-of-Experts does not establish that Soup-of-Experts gives better specialists; it may only reflect that each baseline specialist was trained on one eighth of the tokens. The per-domain curves in Fig. 9 appear still decreasing at 17B tokens, so there is no evidence of saturation. Since CRISP is precisely the method that directly optimizes a model on mix(h_spe), if CRISP trained to 134B tokens beats Soup-of-Experts, the paper's central empirical claim ('achieves the best performance among all methods on the specialized domains') is false; if not, the confound is resolved. This is a correctness risk in the reported comparison, not a disagreement with the method's plausibility.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Soup-of-Experts (SoE), an asymmetrical architecture in which a small model is instantiated as a linear combination of a shared parameter vector S and n expert vectors E_j, with combination coefficients produced by a small MLP applied to domain-mixture weights h. Pretraining samples h from a meta-distribution π, instantiates Θ = S + Σ_j α_j E_j, and optimizes the next-token prediction loss on a batch drawn from mix(h). At specialization time, a target dataset is summarized into weights h_spe via the nearest-centroid method (Algorithm 3), and Θ = Combine(S,E,φ(h_spe)) is used without further training. The paper evaluates on 16 Pile domains with 110M-parameter GPT-2 models, comparing against generic pretraining, per-domain experts, and CRISP, and reports ablations on model scale, support size s, low-rank experts, and number of samples for weight estimation. The central claim is that SoE yields the best specialists while staying close to generic pretraining on the generic loss.","tokens_in":16408,"tokens_out":5958,"duration_ms":54646,"significance":"Training a parameter bank whose linear combinations can be selected by a learned function of domain weights is a novel and practically relevant idea, and the evaluation design is mostly sound: the 16 Pile specialization domains and their estimated weights are held out during SoE pretraining, and the reported losses are measured on those held-out domains. The algorithms are described precisely enough to be reimplemented, and the ablations on support size, low-rank experts, and model scale give useful guidance. The main weakness is the unequal training budget in the headline comparison, which currently prevents the paper from supporting its strongest empirical claim.","major_comments":[{"comment":"The headline comparison is confounded by unequal training budgets. Appendix A states that the Soup-of-Experts and generic pretraining run for 1024K iterations (134B tokens), while domain experts and CRISP run for only 128K iterations (17B tokens) per model. Since Figure 4 plots loss against training time, the baselines are evaluated after one eighth of the tokens used by SoE. Figure 9 shows the per-domain CRISP and domain-expert curves still decreasing at 17B tokens, so there is no evidence of saturation. Consequently, the claim in Section 3.2 that SoE 'achieves the best performance among all methods on the specialized domains' is not established by the reported experiment. I request a token-matched comparison: either train CRISP and the domain experts to 134B tokens, or report SoE losses at 17B tokens alongside the baselines at 17B tokens.","section":"Appendix A, Figure 4"},{"comment":"The description of the x-axis in Figure 4 is ambiguous and possibly inconsistent. The text says that for the multi-model baselines the reported time is the total training time taken by all models, but the CRISP and domain-expert curves appear to lie in the same 0-20h range as SoE, which would not be the case if 16 CRISP models and 64 domain experts were aggregated (each baseline model runs only 128K iterations). Please state clearly whether the x-axis for each baseline is per-model time or total time, and make the comparison readable under both conventions.","section":"Section 3.2"}],"minor_comments":[{"comment":"The phrase '128K iterations (134B tokens)' is arithmetically inconsistent with the stated batch size (128 sequences of length 1024), which gives approximately 17B tokens; this appears to be a typo that should be corrected.","section":"Appendix A"},{"comment":"The main experiments use a meta-distribution support size of s=4, while the ablation reports the lowest average specialized loss at s=2; please justify the choice of s=4 or state explicitly that the difference from the optimal value is small.","section":"Section 3.4, Figure 7"},{"comment":"The paper does not report variance or multiple seeds; given the small differences between some curves (e.g., Figure 6), a statement about run-to-run variability would strengthen the empirical claims.","section":"Section 3"},{"comment":"Algorithm 3 refers to 'Bert(x)' without specifying the exact BERT model or preprocessing used to compute document embeddings; please add this detail or a pointer to the implementation.","section":"Algorithm 3"},{"comment":"There are several typos: 'proportionnal' in the Figure 4 caption, 'an domain weights' in the Figure 2 caption, and 'litterature' in Section 4; these should be fixed in a revision.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper comes from a group that includes the authors of CRISP (Grangier et al., 2024b), which is the main baseline and the source of the weight-estimation method. The baseline comparison is fair in design but the budget asymmetry places the baselines at a serious disadvantage; I would urge the editor to require the token-matched comparison before publication. The discussion of the relationship to (Dimitriadis et al., 2023) is adequate and does not raise novelty concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take on arXiv:2502.01804. The central claim — that Soup-of-Experts gives the best specialists — is not actually supported by the experiments, because the comparison is unfair. SoE and generic pretraining get 1024K iterations (134B tokens), while CRISP and domain experts get 128K iterations (17B tokens). That's an eightfold gap. Figure 9 shows the per-domain baseline curves still going down at 17B tokens, so there's no evidence they'd stay below SoE if given the same budget. This is a real confound, not a nitpick.\n\nThat said, the idea is new and worth engaging. Jointly pretraining a shared base plus a set of expert parameters, with a small MLP mapping domain-weight vectors to combination coefficients, is not model soup or Pareto manifold learning as previously formulated. The training loop — sampling random domain weights and backpropping through the instantiated model — is clean. And the evaluation is largely honest: the 16 Pile domains and their estimated weights are held out during SoE pretraining, so there's no circularity in the specialization metric. The ablations are informative, especially the support-size study (s=2 best for specialized loss) and the low-rank experts analysis. The sample-efficiency result — a handful of samples suffice to get past generic — is a nice practical point.\n\nThe soft spots, in order: (1) the budget confound above; (2) no repeated seeds or error bars, so we don't know if the differences are stable; (3) no code release; (4) no direct average-of-domain-experts baseline, which would be a natural model-merging reference point; (5) the 'instantiate in a flash' procedure depends on the BERT nearest-centroid weight estimator, and there's no test of what happens when the target distribution lies off the simplex of pretraining domains. That last one is a caveat, not a fatal flaw.\n\nNet: the paper deserves a serious referee. The architecture is a real contribution, and the limitations are addressable in revision. But the central empirical claim should be re-run with matched budgets, and ideally with error bars, before anyone takes 'best specialists' at face value. I'd send it to review, expecting substantial revision.","headline":"The architecture is genuinely novel and worth citing, but the headline empirical claim is confounded by an eightfold training-budget gap between Soup-of-Experts and its baselines, so 'best specialists' is not established.","tokens_in":16960,"tokens_out":2409,"would_cite":true,"duration_ms":22418,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single pretrained parameter bank can instantly instantiate a specialist model for any domain mixture.","keywords":["model merging","parameter averaging","domain specialization","language modeling","importance sampling","mixture of experts","low-rank adaptation","task-adaptive pretraining"],"falsifier":"Construct a specialization set whose documents are systematically outside the pretraining domain clusters, for example a language or file format absent from RedPajama2; compute $h_{\\text{spe}}$ with Algorithm 3, instantiate the soup without training, and compare against (a) generic pretraining, (b) a CRISP model trained on $\\text{mix}(h_{\\text{spe}})$, and (c) the same soup with $\\alpha$ optimized by gradient descent on the target data. If the flash-instantiated model fails to beat generic pretraining while the gradient-optimized $\\alpha$ succeeds, the representation assumption collapses.","tokens_in":15987,"feed_emoji":"⚡","tokens_out":4600,"duration_ms":44908,"temperature":0.7,"pith_summary":"The paper tries to show that one pretraining run can encode a family of small specialist models, so that when a target dataset arrives, a specialist can be obtained by merging parameters rather than retraining. The core claim is that with a shared parameter bank $S$, expert vectors $E_1,\\dots,E_n$, and coefficients $\\alpha=\\phi_\\omega(h)$ produced by a small MLP from domain weights $h$, the instantiated model $\\theta = S + \\sum_j \\alpha_j E_j$ can be trained once to have low loss on every mixture $\\text{mix}(h)$. In experiments on 110M-parameter GPT-2 style models over 4096 RedPajama2 domains and 16 Pile domains, the soup achieves the best average specialized loss among generic pretraining, per-domain experts, and CRISP, while staying close to generic pretraining on the generic loss. If correct, pretraining cost no longer scales with the number of specialization tasks; specialization becomes a quick forward pass through the router and a parameter sum.","feed_headline":"Model bank merges experts into any specialist in a flash","feed_subtitle":"One pretraining run yields many small specialists by averaging expert weights, no retraining needed.","key_machinery":"The load-bearing object is the linear combination operator $\\text{Combine}(S,E,\\alpha) = S + \\sum_j \\alpha_j E_j$, together with the learned router $\\phi_\\omega: \\mathbb{R}^k \\to \\mathbb{R}^n$ that maps domain weights $h$ to expert coefficients $\\alpha$. The training objective is $L(S,E,\\omega) = \\mathbb{E}_{h\\sim\\pi}[\\mathbb{E}_{x\\sim\\text{mix}(h)}[\\ell(\\text{Combine}(S,E,\\phi_\\omega(h)), x)]]$, with $\\pi$ a meta-distribution over domain weights that in the main experiments samples $s=4$ sparse domains uniformly. This mechanism carries the argument because it turns model merging from a post-hoc fine-tuning operation into a pretraining objective: the experts are jointly optimized so that their linear interpolation, not any individual expert, yields good models across the entire simplex of mixtures.","core_discovery":"The central claim is that pretraining can amortize specialization: instead of training one model per domain mixture, learn a parameterized family of models via parameter averaging. For any non-negative domain weights $h$ summing to one, the model $\\text{Combine}(S,E,\\phi_\\omega(h)) = S + \\sum_{j=1}^n \\alpha_j(h) E_j$ should have low next-token loss on data sampled from $\\text{mix}(h)$, where $\\phi_\\omega$ is a two-layer MLP mapping histograms over pretraining domains to expert coefficients. Training samples random domain weights $h \\sim \\pi$, instantiates the corresponding model, samples a mini-batch from $\\text{mix}(h)$, and backpropagates through the loss. At specialization time, a target dataset $D_{\\text{spe}}$ is mapped to domain weights $h_{\\text{spe}}$ using the nearest-centroid rule of Algorithm 3, fed through the MLP, and the resulting parameters are merged directly, with no optimization on the target data. The paper reports that this procedure yields the best specialized loss among the compared methods on 16 Pile domains, nearly matches generic pretraining on the generic loss, and preserves its advantage through fine-tuning.","pith_inferences":["The flash specialization procedure inherits its weakness from the domain-weight representation: if a target distribution lies far from the pretraining simplex, the nearest-centroid estimate $h_{\\text{spe}}$ may be meaningless, and the method has no built-in correction because $\\alpha$ is never adapted from target samples.","The router is trained only on pretraining mixtures, so its predictions for out-of-simplex or novel target mixtures are extrapolations; a testable extension is to train the router on a richer meta-distribution or to add a second stage that optimizes $\\alpha$ directly on the target when compute allows.","The soup trades storage flexibility for instant adaptability: a single 14B-parameter soup stores the same total parameter count as 128 individual 110M experts, but it can serve any mixture without knowing the target in advance, which matters when many specialists must be shipped quickly.","The linear-combination subspace may transfer to other modalities or tasks where data domains cluster, since the method only requires a domain-sampling procedure and a loss, not language-specific machinery."],"forward_implications":["A single pretrained soup can instantiate a different specialist for each incoming dataset with no gradient updates, so specialization cost becomes a forward pass through a small MLP plus a parameter sum.","Pretraining cost is decoupled from the number of specialization tasks: CRISP and domain experts scale one pretraining run per task or domain, while the soup does not.","The soup's advantage survives fine-tuning: fine-tuning the instantiated specialist keeps the gain over fine-tuning a generic model, saving up to 10 million fine-tuning tokens on the uspto domain in the reported experiments.","Specialization data can be scarce: with roughly three target samples the instantiated soup already matches generic pretraining, and 100 to 1000 samples are enough to reach the full benefit of the method.","Low-rank experts can reduce the total parameter count, but at fixed parameter budget dense experts are more parameter-efficient; low-rank experts remain a fallback for heavily resource-constrained settings."],"supporting_citations":[{"why":"Supplies the CRISP importance-sampling baseline and the Algorithm 3 nearest-centroid rule used to estimate specialist domain weights from a target dataset.","marker":"(Grangier et al., 2024b)"},{"why":"Provides the model-soup observation that fine-tuned models can be linearly averaged, the key premise that motivates the linear combination of experts.","marker":"(Wortsman et al., 2022)"},{"why":"Provides the RedPajama2 pretraining corpus that is clustered into the 4096 pretraining domains.","marker":"(Weber et al., 2024)"},{"why":"Provides the Pile dataset whose 16 domains are used as specialization targets and evaluation sets.","marker":"(Gao et al., 2020)"},{"why":"Defines the per-domain expert baseline that trains one small model per pretraining domain.","marker":"(Gross et al., 2017)"},{"why":"Supplies the LoRA low-rank parameterization used for the low-rank expert ablation.","marker":"(Hu et al., 2021)"},{"why":"Is the closest prior architecture that dynamically mixes expert weights by task, and the paper contrasts its training loop and parameter sharing with it in Appendix B.","marker":"(Dimitriadis et al., 2023)"}],"fun_headline_variants":["Averaging experts yields any specialist model on demand","Pretrain once, instantiate any specialist via weight averaging","Parameter averaging creates instant specialists without retraining","Soup-of-Experts: blend expert weights for domain-specific models","Instant domain-specific models from a pretrained expert bank"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"A target specialization set can be faithfully summarized by a mixture-weight vector $h_{\\text{spe}}$ computed with the BERT nearest-centroid rule, and the router trained on pretraining mixtures will produce good coefficients for that vector without ever seeing target samples.","fun_headline_variants_meta":{"raw":{"variants":["Averaging experts yields any specialist model on demand","Pretrain once, instantiate any specialist via weight averaging","Parameter averaging creates instant specialists without retraining","Soup-of-Experts: blend expert weights for domain-specific models","Instant domain-specific models from a pretrained expert bank"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1483,"prompt_tokens":929,"completion_tokens":554,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":477}},"tokens_in":545,"tokens_out":554,"duration_ms":5228,"temperature":1.0,"reasoning_tokens":477,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T14:23:48.524982+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a specialization set whose documents are systematically outside the pretraining domain clusters, for example a language or file format absent from RedPajama2; compute $h_{\\text{spe}}$ with Algorithm 3, instantiate the soup without training, and compare against (a) generic pretraining, (b) a CRISP model trained on $\\text{mix}(h_{\\text{spe}})$, and (c) the same soup with $\\alpha$ optimized by gradient descent on the target data. If the flash-instantiated model fails to beat generic pretraining while the gradient-optimized $\\alpha$ succeeds, the representation assumption collapses.","supporting_citations":[{"cited_title":"Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A","cited_arxiv_id":null,"evidence_quote":"Provides the model-soup observation that fine-tuned models can be linearly averaged, the key premise that motivates the linear combination of experts."},{"cited_title":"Hard mixtures of experts for large scale weakly supervised vision","cited_arxiv_id":null,"evidence_quote":"Defines the per-domain expert baseline that trains one small model per pretraining domain."},{"cited_title":"Pareto manifold learning: Tackling multiple tasks via ensembles of single-task models","cited_arxiv_id":null,"evidence_quote":"Is the closest prior architecture that dynamically mixes expert weights by task, and the paper contrasts its training loop and parameter sharing with it in Appendix B."}],"review_version":1}