StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillation

Bryan Bo Cao; Ranjith Merugu; Shubham Jain

REVIEW 6 major objections 5 minor 27 references

StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillation

T0 review · 6 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read StatsMerging claims that a small neural network can merge eight task-specific ViT-B/32 models into one that reaches 94.5% average accuracy, using only weight statistics and teacher-distilled pseudo-labels, without ground-truth labels or…

desk verdict The headline 94.5% comes from an ill-specified StatsMerging++ variant; the base StatsMerging scores 84.5%, below WEMoE's 89.4%, and the paper overclaims both first-ness and heterogeneous merging. read the letter →

arxiv 2506.04567 v1 pith:TTVNHS6Q submitted 2025-06-05 cs.LG cs.CV

Ranjith Merugu , Bryan Bo Cao , Shubham Jain This is my paper

classification cs.LGcs.CV

keywords modelmergingweightstatisticssingularvaluedecompositiontask-specificteacherdistillationmulti-tasklearningvisiontransformertaskvectorsknowledge

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

StatsMerging claims that the right signal for merging several fine-tuned vision models is the distribution of their weights, not the raw weights or hand-tuned coefficients. A lightweight MLP, StatsMergeLearner, takes four statistics of each task model's weights — mean, variance, magnitude, and the leading singular values from SVD — and predicts per-layer merging coefficients. The learner is trained by task-specific teacher distillation: each task model labels unlabeled validation images, and the merged model is trained to match those pseudo-labels, so no ground-truth annotations or test samples enter the loop. On eight ViT-B/32 classification tasks, the layer-wise extension StatsMerging++ achieves 94.5% average accuracy, beating WEMoE (89.4%) by 5.1%, and the paper claims this is the first merging method to handle heterogeneous architectures, by distilling different backbones into a single target architecture before merging.

What carries the argument

The load-bearing object is StatsMergeLearner, a compact two-layer MLP that predicts merging coefficients $\lambda$ from a weight-statistics vector. The vector is $[\mu, \sigma^2, m, \sigma'_r]$: mean, variance, L2 magnitude, and top-$r$ singular values from SVD of the weight matrices; the singular values are the distinctive component, acting as a proxy for task importance and weight distribution, and the ablation attributes +3.0% (same architecture) and +3.2% (different architectures) average accuracy to them. Training relies on task-specific teacher distillation: each fine-tuned task model is a teacher that pseudo-labels unlabeled validation images, and StatsMergeLearner is fit so the merged model matches those one-hot labels under cross-entropy. For heterogeneous architectures, the machinery adds a preprocessing step in which every task model is distilled into one selected target backbone before statistics are extracted and coefficients are predicted.

What would settle it

Run the same eight-task merging experiment but train StatsMergeLearner using pseudo-labels generated by deliberately broken teachers, for example models with shuffled or randomly reinitialized final layers, on the same validation images; if merged accuracy remains near 94.5%, teacher quality is not what carries the result, and the claimed mechanism is not the active one. A second check is to hold out one task entirely and use the learner trained on the remaining seven to predict its coefficients; a large accuracy collapse on the held-out task would falsify the generalization claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that weight-distribution statistics, and especially singular values, are a sufficient guide for predicting how much each task model should contribute to a merged model. For each task $k$, StatsMerging builds a statistics vector $S_k = \mathrm{stats}(\theta_k) = [\mu, \sigma^2, m, \sigma'_r]$, where $\mu$ and $\sigma^2$ are the mean and variance of the weights, $m$ is the weight magnitude, and $\sigma'_r$ are the top $r$ singular values from SVD of the weight matrices. A two-layer MLP, StatsMergeLearner, maps $S_k$ to merging coefficients $\lambda_k$ at task or layer granularity, and the merged weights are assembled in the task-vector form $\theta_{\mathrm{merged}} = \theta_{\mathrm{pre}} + \sum_k \lambda_k (\theta_k - \theta_{\mathrm{pre}})$. The learner is trained by task-specific teacher distillation: each task-specific model pseudo-labels its own unlabeled validation images, and the merged model is optimized with cross-entropy against those pseudo-labels, so no ground-truth annotations or test samples are needed. On eight ViT-B/32 tasks the layer-wise StatsMerging++ reaches 94.5% average accuracy versus 89.4% for WEMoE, and the paper claims the same machinery works for heterogeneous architectures by first distilling all backbones into one target architecture.

Load-bearing premise

The load-bearing premise is that unlabeled validation images for each task are available and that the task-specific teachers' pseudo-labels on those images are accurate enough to train the coefficient predictor; the paper itself restricts its claims to vision classification tasks, so if the validation images are unrepresentative, or the teachers are unreliable, the predicted coefficients and the reported gains will not transfer.

Editorial extensions

If this is right

A single merged ViT-B/32 can replace eight separately fine-tuned models for SUN397, Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, and DTD, holding 94.5% average accuracy versus 90.5% for the individually trained models.
Merging no longer needs ground-truth labels or test samples, only unlabeled validation images and the existing task teachers, which removes the main supervision bottleneck in learning-based merging.
Because coefficients are inferred per layer from weight statistics, the predictor can be applied at test time to new task models without retraining the learner, provided the same statistics are available.
Heterogeneous architectures can be merged by first distilling all task models into a shared backbone; the paper reports 81.3% average accuracy across ResNet50 and ViT-B/32 models, beating Task Arithmetic's 73.7%.
Robustness to image corruption improves: under impulse noise, StatsMerging gains 6.3 percentage points over AdaMerging on a four-task suite.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: if singular values really encode task importance, the same statistics-to-coefficient map might transfer to a ninth unseen task without retraining StatsMergeLearner; the paper tests generalization to two unseen tasks but does not test this zero-shot coefficient transfer.
Editorial inference: the paper explicitly limits itself to vision classification, so applying the same weight-statistics recipe to detection, restoration, or language models is an open question, since their weight distributions and teacher reliability differ.
Editorial inference: a natural stress test would be to train StatsMergeLearner on a few tasks and apply it to many more; the sparse, layer-recurring coefficient patterns the paper visualizes suggest the learner may be capturing architecture structure (attention blocks, MLP layers, norms) more than task identity, which would imply broad transfer.
Editorial inference: the teacher-distillation setup could be tightened by ablating teacher quality, for example by corrupting a fraction of pseudo-labels, to measure how much of the 94.5% depends on teacher accuracy versus the statistics features themselves.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The headline number is the first thing to check: the 94.5% average accuracy that drives the abstract and conclusion belongs to StatsMerging++, a variant described only as “trained on more validation data.” The actual proposed method, LW StatsMerging, gets 84.5% in Table 2, which is 4.9 points below WEMoE (89.4%). Read the main table before trusting the abstract.

The core idea has merit. Using weight statistics (mean, variance, magnitude, and SVD singular values) as input to a lightweight MLP that predicts merging coefficients is a plausible design, and the teacher-distillation training scheme is a sensible way to avoid manual labels. The ablation in Table 6 is internally consistent: each statistic contributes, and SVD adds 3.0–3.2% accuracy. That gives some confidence the feature design isn't arbitrary.

The problems are real and load-bearing. Most importantly, the central SOTA claim is not reproducible: StatsMerging++'s validation data budget is never quantified, and several datasets (MNIST, SVHN, GTSRB, Stanford Cars) have no official validation split, so it's unclear whether the method is carving into training data. That could give it a large, hidden data advantage over data-free WEMoE and explain the anomalously large per-task gains on SUN397 and Cars. Second, Algorithm 1's merge formula (θ_merged = Σ λ_k θ_k) doesn't match the task-vector diagram (θ_pre + Σ λ_k T_k); the paper never clarifies which is actually used. Third, the “first heterogeneous architectural merging” claim is an overreach: the authors distill ViT teachers into a ResNet student and then merge homogeneous models, and they cite Adamms (Du et al., 2025) without discussing it. There's also a stray sentence in Appendix B.2 about “regularization samples” that reads like leftover from another draft.

Who should read this? Practitioners curious about learned coefficient merging might find the feature idea worth testing, but as written, the results cannot be trusted and the method's actual advantage over WEMoE is unestablished. I would still send it to peer review, because the approach is plausible and the reported gap, if real, matters; but the referee should demand exact splits, seeds, and a full specification of StatsMerging++ before any acceptance.

Referee Report

6 major / 5 minor

Summary. The paper proposes StatsMerging, a model merging method that predicts per-task (or per-layer) merging coefficients with a small MLP, StatsMergeLearner, whose input is a vector of weight statistics (mean, variance, norm, and top-3 singular values). The learner is trained by distilling pseudo-labels from the task-specific fine-tuned models themselves on validation images, avoiding human annotations. The authors report results on eight ViT-B/32 classification tasks (Table 2), generalization to unseen tasks (Table 3), and a heterogeneous-architecture variant where ViT teachers are distilled into a ResNet50 before merging (Table 4). The headline claim is 94.5% average accuracy for LW StatsMerging++, 5.1% above WEMoE.

Significance. The proposed direction of using weight-distribution statistics to predict merging coefficients and training the predictor by teacher distillation is interesting and, if properly validated, would be a useful addition to the model-merging literature. The paper ships code and includes ablations (Tables 5 and 6) that support the incremental value of the statistical features, and the unseen-task generalization experiments in Table 3 are a relevant extension beyond the standard benchmark. The main concerns are empirical reproducibility and the fairness of the comparison: the state-of-the-art claim rests on StatsMerging++, whose extra validation data are never quantified, and the base method in Table 2 is 4.9 points below WEMoE. These issues must be resolved before the central claim can be assessed.

major comments (6)

[Abstract, Section 4.2, Table 2, Appendix A] The abstract, Section 4.2, and Section 5 claim that StatsMerging outperforms WEMoE by 5.1% (94.5% vs. 89.4%), but Table 2 reports the method actually described in Section 3 and Algorithm 1, LW StatsMerging, at 84.5%, which is 4.9 points below WEMoE. The 94.5% figure comes from LW StatsMerging++, which Appendix A defines only as "an extended version of StatsMerging trained on more validation data." No sample counts, split provenance, or overlap checks are given for this variant, and the paper does not state its training budget. Because the central SOTA claim is defined by an undisclosed data quantity, the result is not reproducible and the comparison to data-free methods such as WEMoE is not interpretable. Please report the exact validation sets and sizes for both StatsMerging and StatsMerging++, and either promote the base method as the headline or justify the extra data as a fair comparison.
[Section 3.1, Algorithm 1, Figure 1] There is an inconsistency in the merging formula. Section 3.1 defines the merged weights as theta_m = sum_k lambda_k theta_k, and Algorithm 1 line 19 writes theta^l_merged = sum_i lambda_k theta_k, but Figure 1(d) and its caption show theta_MTL = theta_pre + lambda_A T_A + lambda_B T_B with T_k = theta_k - theta_pre. If the experimental implementation uses the task-vector form, the text and Algorithm 1 are wrong; if it uses the raw-weight form, the diagram is misleading. The two formulas produce different merged models for the same lambda, so this ambiguity affects every reported number. Please state the exact merging equation used in the code and align Algorithm 1, Figure 1, and Section 3.1.
[Section 3.3, Appendix A, Table 2] Section 3.3 states that all SML training samples are collected from the validation set only, but the paper never defines a validation split per dataset. Of the eight datasets, MNIST, SVHN, GTSRB, and Stanford Cars do not have official validation splits in the cited versions, and Appendix A does not say whether the validation set was carved from training data or includes test images. If any test images were used to train SML, the reported accuracies, especially the large per-task gains in Table 2 (e.g., SUN397 75.3 to 92.4 and Cars 77.7 to 95.4), would be inflated. Please provide per-dataset validation construction details and an overlap check with the test sets; the claim "without test samples" in the abstract needs this evidence.
[Section 3.3, Table 2] The coefficients lambda are learned on validation samples drawn from the same task distributions on which the merged model is tested, and the pseudo-labels come from the very models being merged. This makes lambda a fitted quantity rather than a prediction from weight statistics alone, so the comparison with data-free baselines such as WEMoE and EMR-MERGING is not apples-to-apples. I do not view this as a tautology, but it is a correctness-risk concern: the paper should report a concrete control, e.g., training SML on a held-out subset of validation data and testing on the remaining validation tasks, or limiting the validation budget to the same size used by AdaMerging's test-time adaptation, to show that the gains are not primarily an artifact of fitting to the target distributions.
[Section 4.2, Eq. (8), Table 4] The heterogeneous-architecture experiment uses the distillation loss in Eq. (8), L = alpha L_CE(y, y_hat) + (1-alpha) T^2 L_KL(...), which requires ground-truth labels y. This contradicts the abstract's claim that StatsMerging works "without requiring ground truth labels" for the heterogeneous setting. In addition, Table 4 compares only against three simple baselines on three tasks; no WEMoE, AdaMerging, or other recent merging method is evaluated in this setting, so the "first heterogeneous architectural merging method" claim is not supported by the experiments reported. Please clarify the label requirement for this extension and add heterogeneous-architecture comparisons with the baselines used in Table 2.
[Tables 2 and 3] No error bars or multiple-seed results are reported. Given that several differences are small (e.g., Table 3: AdaMerging++ 68.7 vs. StatsMerging 70.8 in the first group; Table 2: TW StatsMerging 76.4 vs. TW AdaMerging++ 73.7), the reader cannot determine whether the gains are statistically significant. Please report means and standard deviations over at least three seeds for the main merging and generalization tables.

minor comments (5)

[Section 4.2 and Appendix B] Table 2 is referred to in the text as "Table 4.2," and the Appendix robustness tables are numbered Table 5 and Table 6, which collide with the main-text Tables 5 and 6. Please renumber all tables consistently.
[Section 3.2] The sentence "Notably, the Equation 3 above is task-wise" appears to refer to Eq. (2); Eq. (3) is introduced later. Please fix the cross-reference.
[Appendix B.2] Appendix B.2 contains an orphaned paragraph beginning "Identifies 'regularization samples'..." that appears to be leftover draft text; it should be removed or fully integrated into the discussion.
[Figure 3] The caption of Figure 3 does not explain how coefficients are normalized or which layer indices correspond to MHSA, MLP, and LayerNorm blocks; please add this information to make the coefficient analysis interpretable.
[Equations (6), (7), and (10)] The relationship between Eq. (6), Eq. (7), and Eq. (10) is unclear: Eq. (6) uses ground-truth labels but Section 3.3 says no labels are used. Please clarify which loss is actually minimized in each experiment.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the method's coefficients are learned by an external distillation objective on validation data, and the StatsMerging++ validation-budget concern is a reproducibility/fairness issue, not a demonstrated circular step.

full rationale

The derivation chain is: compute deterministic statistics of each task model's weights; feed them to a small MLP (StatsMergeLearner) to produce per-layer coefficients; form the merged model as theta_pre + sum(lambda_k (theta_k - theta_pre)); train the MLP by matching merged-model predictions to task-teacher pseudo-labels on validation images; then report held-out test accuracy. None of these steps defines the test result in terms of the training signal by construction. The teacher pseudo-labels are produced by the same task models being merged, but that is a standard distillation objective, not an identity with the reported test accuracy. The main SOTA claim relies on StatsMerging++ (Appendix A), whose only described difference is 'an extended version of StatsMerging trained on more validation data'; this raises a legitimate reproducibility and comparison-fairness concern, but the paper does not state that validation data were drawn from the test sets or that the extra validation budget was selected on test accuracy, so under the hard rules this is not a demonstrated circularity. Self-citations (e.g., Cao et al. 2024/2025, Merugu et al. 2025) are contextual and not load-bearing. No equation reduces to its own input, and no fitted parameter is renamed as a prediction in a way that makes the empirical claim forced. The central claim is therefore not circular, though the undisclosed validation budget of StatsMerging++ should be flagged as a correctness/reproducibility risk rather than a circularity risk.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central claim depends on a learned coefficient predictor (SML), several hand-chosen hyperparameters (SVD rank 3, T=4.0, alpha=0.7), and assumptions that linear weight interpolation works and that teacher pseudo-labels on validation data are reliable. No formal verification or external benchmark validates these components.

free parameters (4)

SVD rank r for singular-value statistics = 3
Section 3.2 states by default we use rank 3 from Sigma_k; this hand-chosen rank determines the stats vector.
Distillation temperature T = 4.0
Used in Eq. (8) for heterogeneous architecture distillation; chosen without sensitivity analysis.
Distillation loss balance alpha = 0.7
In Eq. (8), L = alpha L_CE + (1-alpha) T^2 L_KL, with alpha=0.7 chosen by hand.
Merging coefficients lambda = Learned; values not reported except as a heatmap
The central coefficients lambda are predicted by StatsMergeLearner and fitted to validation pseudo-labels, not derived from first principles.

assumptions (5)

domain assumption A linear combination of task model weights can represent a high-performing multi-task model.
Section 3.1 defines merging as theta_m = sum lambda_k theta_k. This is the standard weight-space interpolation assumption.
domain assumption Task-specific teachers provide pseudo-labels reliable enough to train the merging learner.
Section 3.3 uses teacher predictions on validation data as supervision; the label-free claim rests on this.
ad hoc to paper The statistics [mean, variance, norm, top-3 singular values] are sufficient to predict good merging coefficients.
Section 3.2 hypothesizes this without derivation; Table 6 shows empirical contributions but no theory.
domain assumption Distilling heterogeneous task models into one common architecture preserves the knowledge needed for merging.
Section 4.2 distills ViT teachers into ResNet students before merging; no analysis of distillation loss is provided.
domain assumption Validation data from each task is representative of that task's test data.
SML is trained on validation samples and evaluated on the same tasks' test sets; any distribution shift would break the reported generalization.

invented entities (1)

StatsMergeLearner (SML)
purpose: A two-layer MLP that maps weight statistics to merging coefficients lambda.
The MLP is only evaluated inside this paper; no pretrained SML weights or commit-hash-reproducible artifacts are provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillation." pith.science (2026). https://pith.science/paper/TTVNHS6Q

@misc{pith2026250604567,
  author       = {Pith},
  title        = {Pith review of: StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TTVNHS6Q}},
  note         = {Machine review of arXiv:2506.04567}
}

read the original abstract

Model merging has emerged as a promising solution to accommodate multiple large models within constrained memory budgets. We present StatsMerging, a novel lightweight learning-based model merging method guided by weight distribution statistics without requiring ground truth labels or test samples. StatsMerging offers three key advantages: (1) It uniquely leverages singular values from singular value decomposition (SVD) to capture task-specific weight distributions, serving as a proxy for task importance to guide task coefficient prediction; (2) It employs a lightweight learner StatsMergeLearner to model the weight distributions of task-specific pre-trained models, improving generalization and enhancing adaptation to unseen samples; (3) It introduces Task-Specific Teacher Distillation for merging vision models with heterogeneous architectures, a merging learning paradigm that avoids costly ground-truth labels by task-specific teacher distillation. Notably, we present two types of knowledge distillation, (a) distilling knowledge from task-specific models to StatsMergeLearner; and (b) distilling knowledge from models with heterogeneous architectures prior to merging. Extensive experiments across eight tasks demonstrate the effectiveness of StatsMerging. Our results show that StatsMerging outperforms state-of-the-art techniques in terms of overall accuracy, generalization to unseen tasks, and robustness to image quality variations.

Figures

Figures reproduced from arXiv: 2506.04567 by the authors.

**Figure 1.** Compared to prior works, StatsMerging uniquely learns the merging coefficients using StatsMergeLearner, taking advantage of statistical features of weigts pre-trained on prior tasks. Notably, while both AdaMerging and StatsMerging are presented in the task-wise level in c) and d) for simplicity of illustration, the same principle can be applied at the layer-wise level for fine-grained adaptation. We make four signif… view at source ↗

**Figure 2.** Knowledge Distillation Diagram. StatsMergeLearner (SML) learns the merging coefficients λ by minimizing the loss between the merged model’s predictions and pseudo labels generated by task-specific teacher models. During inference, only the merged model in StatsMerging is used to predict class labels. Our key intuition is that each pre-trained model θk is already good at its own task dataset {xi , yi}k ∈ Dk, therefor… view at source ↗

**Figure 3.** Heatmap of StatsMerging merging coefficients λ of ViT-B/32 (4) across eight tasks. X-axis: layer index. Y-axis: Tasks. Coefficients are normalized to sum to 1. 5 Conclusion Model merging offers a compelling post-hoc advantage to reduce memory storage from a corpus of large pre-trained models. We propose StatsMerging, a novel merging technique guided by model weight statistical features learned through Task-Specific … view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: StatsMerging++ Training Accuracy Curve. B.5 Future Work and Limitations In this work, we focus on vision-based classification tasks, leaving extensions to other domains, such as object detection (Tan et al., 2020), super-resolution (Sun et al., 2022), and image and vid…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 10 canonical work pages

[3]

Gong Cheng, Junwei Han, and Xiaoqiang Lu

URL https://arxiv.org/abs/2408.12105. Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art.Proceedings of the IEEE, 105(10):1865–1883,

arXiv
[4]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Georgiou, et al. An image is worth 16x16 words: Transformers for image recognition at scale. 2021a. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, ...

arXiv
[5]

Adamms: Model merging for heterogeneous multimodal large language models with unsupervised coefficient optimization.arXiv preprint arXiv:2503.23733,

Yiyang Du, Xiaochen Wang, Chi Chen, Jiabo Ye, Yiru Wang, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Zhifang Sui, et al. Adamms: Model merging for heterogeneous multimodal large language models with unsupervised coefficient optimization.arXiv preprint arXiv:2503.23733,

arXiv
[10]

Manning, Christopher Potts, and Róbert Csordás

Julie Kallini, Shikhar Murty, Christopher D. Manning, Christopher Potts, and Róbert Csordás. Mrt5: Dynamic token merging for efficient byte-level language models. InProceedings of the 13th International Conference on Learning Representations (ICLR 2025),

work page 2025
[11]

MergeDistill: Merging Pre-trained Language Models using Distillation

URL https://openreview.net/forum?id=VYWBMq1L7H. Simran Khanuja, Melvin Johnson, and Partha Talukdar. Mergedistill: Merging pre-trained language models using distillation.arXiv preprint arXiv:2106.02834,

work page Pith review arXiv
[12]

LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging

Jinuk Kim, Marwa El Halabi, Mingi Ji, and Hyun Oh Song. Layermerge: neural network depth compression through layer pruning and merging.arXiv preprint arXiv:2406.12837,

work page Pith review arXiv
[17]

Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte

URLhttps://arxiv.org/abs/2406.07529. Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. InProceedings of the IEEE/CVF international conference on computer vision, pages 1833–1844,

arXiv
[19]

Joint Flow And Feature Refinement Using Attention For Video Restoration

Ranjith Merugu, Mohammad Sameer Suhail, Akshay P Sarashetti, Venkata Bharath Reddy Reddem, Pankaj Ku- mar Bajpai, and Amit Satish Unde. Joint flow and feature refinement using attention for video restoration. arXiv preprint arXiv:2505.16434,

work page Pith review arXiv

Show all 27 references

[20]

Reading digits in natural images with unsupervised feature learning

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. InNIPS workshop on deep learning and unsupervised feature learning, volume 2011, page

2011
[21]

Very deep convolutional networks for large-scale image recognition

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556,

arXiv
[22]

Hierarchical context merging: Better long context understanding for pre-trained llms.arXiv preprint arXiv:2404.10308,

Woomin Song, Seunghyuk Oh, Sangwoo Mo, Jaehyung Kim, Sukmin Yun, Jung-Woo Ha, and Jinwoo Shin. Hierarchical context merging: Better long context understanding for pre-trained llms.arXiv preprint arXiv:2404.10308,

arXiv
[23]

The german traffic sign recognition benchmark: a multi-class classification competition

Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. InThe 2011 international joint conference on neural networks, pages 1453–1460. IEEE,

2011
[24]

Cat merging: A training-free approach for resolving conflicts in model merging.arXiv preprint arXiv:2505.06977,

Wenju Sun, Qingyong Li, Yangli-ao Geng, and Boyang Li. Cat merging: A training-free approach for resolving conflicts in model merging.arXiv preprint arXiv:2505.06977,

arXiv
[25]

Merging models on the fly without retraining: A sequential approach to scalable continual model merging.arXiv preprint arXiv:2501.09522,

Anke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Merging models on the fly without retraining: A sequential approach to scalable continual model merging.arXiv preprint arXiv:2501.09522,

arXiv
[26]

A survey on post-training of large language models.arXiv preprint arXiv:2503.06072,

Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang, Jiangyue Yan, Yao Su, et al. A survey on post-training of large language models.arXiv preprint arXiv:2503.06072,

arXiv
[28]

Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities.arXiv preprint arXiv:2408.07666, 2024a

Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities.arXiv preprint arXiv:2408.07666, 2024a. Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaoju...

arXiv
[1998]

Mitigating parameter interference in model merging via sharpness-aware fine-tuning.arXiv preprint arXiv:2504.14662,

Yeoreum Lee, Jinwook Jung, and Sungyong Baik. Mitigating parameter interference in model merging via sharpness-aware fine-tuning.arXiv preprint arXiv:2504.14662,

arXiv
[2009]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton

URLhttps://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. InAdvances in neural information processing systems, volume 25, pages 1097–1105,

2009
[2013]

doi: 10.1109/ICCVW.2013.77

ISBN 978-1-4799-3022-7. doi: 10.1109/ICCVW.2013.77. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto,

2013 doi
[2015]

Denoising diffusion probabilistic models

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. InAdvances in Neural Information Processing Systems 33 (NeurIPS 2020), pages 6840–6851,

2020
[2016]

Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36:7093–7115, 2023a

Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36:7093–7115, 2023a. Sachin Yadav, Chitta Malaviya, Graham Neubig, and Puneet Agarwal. Mergin...

arXiv
[2018]

Generative adversarial nets

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. InAdvances in neural information processing systems 27 (NIPS 2014), pages 2672–2680,

2014
[2019]

Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

arXiv
[2022]

Merge3: Efficient evolutionary merging on consumer-grade gpus.arXiv preprint arXiv:2502.10436,

Tommaso Mencattini, Adrian Robert Minut, Donato Crisostomi, Andrea Santilli, and Emanuele Rodola. Merge3: Efficient evolutionary merging on consumer-grade gpus.arXiv preprint arXiv:2502.10436,

arXiv
[2023]

Averaging weights leads to wider optima and better generalization.arXiv preprint arXiv:1803.05407,

Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization.arXiv preprint arXiv:1803.05407,

arXiv
[2024]

Data-side efficiencies for lightweight convolutional neural networks.arXiv preprint arXiv:2308.13057,

Bryan Bo Cao, Lawrence O’Gorman, Michael Coss, and Shubham Jain. Data-side efficiencies for lightweight convolutional neural networks.arXiv preprint arXiv:2308.13057,

arXiv
[2025]

I Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, Chun-Yi Lee, et al

URL https://openreview.net/forum?id= 2ET561DyPe. I Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, Chun-Yi Lee, et al. Retraining-free merging of sparse mixture-of-experts via hierarchical clustering.arXiv preprint arXiv:2410.08589,

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [3]

Gong Cheng, Junwei Han, and Xiaoqiang Lu

URL https://arxiv.org/abs/2408.12105. Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art.Proceedings of the IEEE, 105(10):1865–1883,

arXiv

[2] [4]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Georgiou, et al. An image is worth 16x16 words: Transformers for image recognition at scale. 2021a. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, ...

arXiv

[3] [5]

Adamms: Model merging for heterogeneous multimodal large language models with unsupervised coefficient optimization.arXiv preprint arXiv:2503.23733,

Yiyang Du, Xiaochen Wang, Chi Chen, Jiabo Ye, Yiru Wang, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Zhifang Sui, et al. Adamms: Model merging for heterogeneous multimodal large language models with unsupervised coefficient optimization.arXiv preprint arXiv:2503.23733,

arXiv

[4] [10]

Manning, Christopher Potts, and Róbert Csordás

Julie Kallini, Shikhar Murty, Christopher D. Manning, Christopher Potts, and Róbert Csordás. Mrt5: Dynamic token merging for efficient byte-level language models. InProceedings of the 13th International Conference on Learning Representations (ICLR 2025),

work page 2025

[5] [11]

MergeDistill: Merging Pre-trained Language Models using Distillation

URL https://openreview.net/forum?id=VYWBMq1L7H. Simran Khanuja, Melvin Johnson, and Partha Talukdar. Mergedistill: Merging pre-trained language models using distillation.arXiv preprint arXiv:2106.02834,

work page Pith review arXiv

[6] [12]

LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging

Jinuk Kim, Marwa El Halabi, Mingi Ji, and Hyun Oh Song. Layermerge: neural network depth compression through layer pruning and merging.arXiv preprint arXiv:2406.12837,

work page Pith review arXiv

[7] [17]

Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte

URLhttps://arxiv.org/abs/2406.07529. Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. InProceedings of the IEEE/CVF international conference on computer vision, pages 1833–1844,

arXiv

[8] [19]

Joint Flow And Feature Refinement Using Attention For Video Restoration

Ranjith Merugu, Mohammad Sameer Suhail, Akshay P Sarashetti, Venkata Bharath Reddy Reddem, Pankaj Ku- mar Bajpai, and Amit Satish Unde. Joint flow and feature refinement using attention for video restoration. arXiv preprint arXiv:2505.16434,

work page Pith review arXiv