{"paper":{"title":"Bayesian Model Merging","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Bayesian Model Merging fuses task-specific models into one via inner Bayesian regression under anchor priors and outer Bayesian optimization of per-module hyperparameters.","cross_cats":["cs.AI"],"primary_cat":"cs.LG","authors_text":"Kaiyang Li, Qing Su, Shaobo Han, Shihao Ji","submitted_at":"2026-05-13T00:36:47Z","abstract_excerpt":"Model merging aims to combine multiple task-specific expert models into a single model without joint retraining, offering a practical alternative to multi-task learning when data access or computational budget is limited. Existing methods, however, face two key limitations: (1) they overlook the valuable inductive bias of strong anchor models and estimate the merged weights from scratch, and (2) they rely on a shared hyperparameter setting across different modules of the network, lacking a global optimization strategy. This paper introduces Bayesian Model Merging (BMM), a plug-and-play bi-leve"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Across extensive benchmarks, including up to 20-task merging in vision and 5-task merging in language, BMM consistently outperforms all plug-and-play anchor baselines (e.g., TA, WUDI-Merging, and TSV). In particular, on the ViT-L/14 benchmark for 8-task merging, a single merged model reaches 95.1, closely matching the average performance of eight task-specific experts (95.8).","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The claimed alignment between activation statistics and task vectors that enables the data-free Gram-matrix estimation, together with the assumption that the inner-level Bayesian regression under the anchor prior produces a solution that generalizes without hidden post-hoc adjustments.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Bayesian Model Merging introduces a bi-level optimization framework that merges task-specific models via closed-form Bayesian regression with an anchor prior and global hyperparameter search, outperforming baselines and nearly matching expert averages on up to 20-task vision and 5-task language Merg","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Bayesian Model Merging fuses task-specific models into one via inner Bayesian regression under anchor priors and outer Bayesian optimization of per-module hyperparameters.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"ab7c2ad3c1596973df5628ea20ea381dffce5e00899de90a989efa72921045f4"},"source":{"id":"2605.12843","kind":"arxiv","version":1},"verdict":{"id":"755c71a3-bfc8-40b5-9114-45252ba5e03b","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-14T20:38:04.112438Z","strongest_claim":"Across extensive benchmarks, including up to 20-task merging in vision and 5-task merging in language, BMM consistently outperforms all plug-and-play anchor baselines (e.g., TA, WUDI-Merging, and TSV). In particular, on the ViT-L/14 benchmark for 8-task merging, a single merged model reaches 95.1, closely matching the average performance of eight task-specific experts (95.8).","one_line_summary":"Bayesian Model Merging introduces a bi-level optimization framework that merges task-specific models via closed-form Bayesian regression with an anchor prior and global hyperparameter search, outperforming baselines and nearly matching expert averages on up to 20-task vision and 5-task language Merg","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The claimed alignment between activation statistics and task vectors that enables the data-free Gram-matrix estimation, together with the assumption that the inner-level Bayesian regression under the anchor prior produces a solution that generalizes without hidden post-hoc adjustments.","pith_extraction_headline":"Bayesian Model Merging fuses task-specific models into one via inner Bayesian regression under anchor priors and outer Bayesian optimization of per-module hyperparameters."},"references":{"count":66,"sample":[{"doi":"","year":2017,"title":"An Overview of Multi-Task Learning in Deep Neural Networks","work_id":"2c849184-06ce-40df-8143-ec29eb925f39","ref_index":1,"cited_arxiv_id":"1706.05098","is_internal_anchor":true},{"doi":"","year":2022,"title":"Matena and Colin A","work_id":"f85bb7a2-7dff-4a41-aa1a-a2013fc51ef4","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2023,"title":"Editing models with task arithmetic","work_id":"a4f3855c-b66f-4915-8b5b-581e494a8cb5","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2023,"title":"Raffel, and Mohit Bansal","work_id":"ac02f012-a7c9-4546-a950-364c16bc8719","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2026,"title":"The hugging face hub","work_id":"051718a0-462f-4455-962a-aa2930577451","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":66,"snapshot_sha256":"507badcd941774027e9060eec0a645288c2ed7293cb430fdf90147a4c0ac252a","internal_anchors":11},"formal_canon":{"evidence_count":1,"snapshot_sha256":"c3a047cd8172fed4c4a0b52f3b10ba64430131dd805daceefaa009725d2019bf"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}