REVIEW 3 major objections 6 minor 25 references
AgentPatch: Coarse-to-Fine Weak-Task Repair for Merging Agentic Multimodal Large Language Models
T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read AgentPatch claims that a merged agentic multimodal model can be repaired, without retraining, to recover its weakest interactive capability while preserving search and visual processing, raising the six-benchmark average from 54.5 to 56.6.
desk verdict AgentPatch is a genuinely new, well-specified repair pipeline for merged agentic MLLMs; the headline gains rest on one small dev partition, so the decisive question is whether the numbers survive a re-split, not whether the method is sound. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a three-stage parameter surgery on task vectors. Stage 1 picks the best training-free merge on fixed development splits. Stage 2 builds the weak-task unique mask $U^p_w = A^p_w \prod_{k \ne w}(1-A^p_k)$, a boolean mask of coordinates active in the weak-task expert but not in any other expert, and applies $\theta^{(2)} = \theta^{(1)} + \beta U_w \odot (\theta^w - \theta^{(1)})$. Stage 3 compiles repair and protection behavior spans from paired recipient-expert trajectories using the Diagnoser-Guardian-Compiler workflow, scores each feed-forward neuron by trajectory-balanced activation times output-projection norm, takes top fractions per behavior group, removes Guardian-protected neurons, and interpolates whole neuron blocks toward the weak-task expert. The key identity is that restricting recovery to unique weak-task coordinates plus excluding protected neurons converts merging from global averaging into targeted, capability-aware repair.
What would settle it
Hold out different development partitions for the same six benchmarks (for example, a different 24 AndroidWorld and 74 OSWorld tasks) and rerun the full AgentPatch pipeline without touching the current test sets; if GUI is no longer selected as the weakest capability or the AndroidWorld and OSWorld gains disappear, the reported repair depends on the specific calibration split.
Extended reading notes
Core claim
The paper claims that the bottleneck in merging agentic MLLMs is not parameter conflict in general but the asymmetric loss of interactive capabilities and the loss of single decisive actions. AgentPatch's central claim is that a merged model can be repaired without retraining: first choose the most stable merge, here TSVM, as recipient; then restore only the weak-task expert's unique residual, coordinates active in GUI-Owl-7B and inactive in the other experts, with coefficient $\beta=0.15$; finally locate feed-forward neurons that fire on expert-success behaviors, exclude neurons tied to recipient-protected behaviors via a Guardian mask, and softly interpolate the remaining neurons toward the GUI expert with coefficient $\alpha=0.02$. The reported result is a single static checkpoint that improves diverse merged backbones, alleviates weak-task degradation, and better balances weak-task recovery with preservation of search and visual capabilities.
Load-bearing premise
The repair targets and all hyperparameters come from small fixed development partitions (24 AndroidWorld and 74 OSWorld tasks, 6.7% to 20.9% of each pool) and from experts built on one shared base model; if those partitions misrepresent which capability is weak or the chosen coefficients do not transfer, the reported test-set gains will not replicate.
Editorial extensions
If this is right
- Merged agentic MLLMs can be repaired into one deployable checkpoint, removing the need for expert routing or ensemble inference.
- Weak-task recovery is non-monotonic: intermediate residual strength ($\beta=0.15$) balances recovery with preservation, while stronger injection ($\beta=0.30$) returns to backbone-level performance.
- Behavior-level repair transfers across environments of the same capability: the same patch improves both AndroidWorld and OSWorld.
- Explicit capability protection is load-bearing: disabling the Guardian exclusion costs 5.0 points on GUI and 0.6 points on visual benchmarks.
- The repair generalizes across merged recipients: applying the same pipeline to TSVM and Iso-CTS raises the six-benchmark average by 2.1 and 1.1 points respectively.
Reading between the lines
- We note that 'training-free' applies to the final checkpoint: the DGC workflow invokes an external analysis model on full trajectories during offline construction, and the paper does not report this one-time cost; a deployment would weigh it against the serving savings of a single checkpoint.
- The unique-residual mask is a hard boolean product; a natural extension is a soft mask weighting coordinates by relative task-vector magnitude, which could reduce sensitivity to the $\beta=0.15$ choice.
- The same coarse-to-fine scheme could be applied recursively: after GUI is repaired, run Stages 2 and 3 with search or visual as the weak task on their own calibration sets to see whether the balance improves further.
- A testable prediction for scaling: adding a fourth expert (for example, a web-browsing agent) should still leave GUI as the repair target as long as the new capability does not dominate the development-split average.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Agentic MLLM Merging as a task and proposes AgentPatch, a training-free, coarse-to-fine pipeline that (i) selects a stable merged backbone among candidate merging operators, (ii) restores weak-task-exclusive residual signals via a parameter mask, and (iii) applies an agent-guided behavior-critical patch that localizes FFN neurons from compiled repair/protection behavior spans and interpolates them toward the weak-task expert. Experiments use Qwen2.5-VL-7B with three experts (search, GUI, visual processing) and six benchmarks, with dev/test partitions kept disjoint. The headline result is an improvement of the TSVM backbone from 54.5 to 56.6 average, with AndroidWorld rising from 59.2 to 63.6 and OSWorld from 17.7 to 19.0, while V*Bench is preserved at 84.8. The paper reports component ablations, design ablations, hyperparameter sensitivity, a second-recipient generalization check, and qualitative trajectory evidence.
Significance. If the reported results hold, the paper makes a useful practical contribution: it shows that a merged agentic MLLM can be repaired to recover a degraded interactive capability without retraining, routing, or ensembles, and it articulates two concrete failure modes (asymmetric capability preservation and behavior-critical forgetting). The evaluation is largely honest: backbone selection, weak-task identification, and hyperparameter tuning are performed on fixed development partitions and the reported Table 1 numbers are on disjoint test partitions, so the evaluation is not circular. The paper also provides clean ablations, a second-recipient generalization check, a detailed DGC evidence contract, and a code release. The main weakness is statistical: all tables report single runs without variance or significance tests on stochastic interactive benchmarks, and the dev partitions used for pipeline decisions are small, so the magnitude and reproducibility of the headline OSWorld gain are not yet established.
major comments (3)
- [Experiments, Table 1] All evaluation tables report single-run scores with no variance, no multiple seeds, and no significance tests. This matters directly for the central claim because AndroidWorld and OSWorld are dynamic, stochastic environments and the headline OSWorld gain is only 1.3 points (17.7 to 19.0), which is close to plausible run-to-run noise. Please provide repeated evaluations (multiple seeds or repeated executions with the same checkpoint) with means and standard deviations, and, where feasible, a paired test across tasks or trajectories to establish that the GUI gains and the overall average gain are not noise.
- [Methodology (Eqs. 5, 6, 9, 15) and Supplementary A.2] The pipeline makes every load-bearing choice on fixed development partitions that are small relative to the test sets: AndroidWorld dev has 24 tasks and OSWorld dev has 74 tasks, and these partitions drive Stage 1 backbone selection, Eq. (6) weak-task identification, Eq. (9) beta, the Stage 3 ratios q_r and q_guard, and Eq. (15) alpha. Because Table 3 shows that the method is sensitive to neuron localization (GUI drops by 5.6 points with random neurons) and Guardian exclusion (GUI drops by 5.0 points without it), dev-set noise can propagate through the entire pipeline. The paper reports no re-split stability check, no bootstrap over dev partitions, and no evidence that the identified weak task and the chosen hyperparameters are stable. Please add a dev-split resampling or stability analysis showing that the GUI weak-task identification and the chosen coefficients are robust to the particular dev sample.
- [Related Work and Table 1] The Related Work section cites ARM (Feng et al. 2026) as the most recent method for merging interactive LLM agents, yet ARM is absent from the baseline comparison in Table 1. Since ARM is the closest existing approach to the paper's setting, its omission means the claim that AgentPatch 'improves diverse merged backbones' relative to the state of the art is not fully established. Please include ARM in Table 1, or explicitly justify its exclusion (for example, if its code or checkpoints are unavailable).
minor comments (6)
- [Table 1] The TIES-Merging row has a formatting error: '44.619.12.0' should show separate values for AndroidWorld, OSWorld, and V*Bench, and the OSWorld score '19.1' appears to be missing from the printed alignment.
- [Figure 5] The figure label 'SeletedBackbone' contains a typo and should read 'SelectedBackbone'.
- [Abstract and Table 4] The abstract's claim that AgentPatch 'improves diverse merged backbones' is supported by only two recipients (TSVM and Iso-CTS) in Table 4. Consider testing additional merging operators as recipients, or tempering the wording to 'multiple merged backbones'.
- [Supplementary A.4] The per-group repair ratios q_r (0.0075 for AndroidWorld and 0.0025 for OSWorld) appear only in the supplementary table; the main text should state at least the value range or refer explicitly to the supplementary table when introducing q_r in Eq. (12).
- [Limitations] The Limitations section acknowledges the offline preprocessing cost and the use of three experts, but does not mention the sensitivity of the method to the dev-partition hyperparameters beta, q_r, q_guard, and alpha; adding a sentence would make the limitations more complete.
- [Equation (14)] The set-difference notation M^ell = (union_r R^ell_r) \ (union_p P^ell_p) is clear, but the text could define R^ell_r and P^ell_p explicitly before Eq. (14) rather than only in the surrounding prose; currently the reader must infer the symbols from the preceding paragraphs.
Circularity Check
No significant circularity: reported gains are measured on test partitions disjoint from backbone selection, weak-task identification, and hyperparameter tuning, and the only self-citations are not load-bearing.
full rationale
The paper's derivation chain is not circular. Stage 1 (Eq. 5) selects a merged backbone by the highest unweighted mean on fixed development subsets, and Eq. 6 identifies the weakest-preserved capability on the same subsets; Stage 2 (Eq. 9) constructs a coarse repair using task-vector masks; Stage 3 (Eqs. 10-15) compiles behavior spans from calibration trajectories that are stated to be disjoint from test tasks (Supplementary A.2, Table 1). The reported Table 1 numbers are test-set results on disjoint partitions, so no reported gain is numerically forced by the fitting procedure. Hyperparameters such as beta=0.15, q_r, q_guard, and alpha are selected on the development partitions, which is ordinary model selection rather than circularity; the small size of those partitions is a generalization-risk concern, not a circularity finding. The self-citations (PivotMerge, EvoMAS) appear only in related-work context and do not carry a load-bearing premise. Although Eq. 15 intentionally interpolates selected neurons toward the weak-task expert, the claim that this improves the merged model is empirical: ablations (Table 3) show that random neuron localization or Guardian removal degrades GUI by 5-6 points, so the outcome is not a by-construction consequence of the update rule. No uniqueness theorem or ansatz is imported from the authors' prior work.
Assumptions & free parameters
free parameters (6)
- Residual coefficient beta =
0.15
- Interpolation coefficient alpha =
0.02
- Repair ratios q_r =
0.0075 for AndroidWorld groups, 0.0025 for OSWorld groups
- Guardian protection ratio q_guard =
0.01
- Activity threshold epsilon =
10^-5
- Task Arithmetic scale lambda_TA =
0.3
assumptions (4)
- domain assumption Expert checkpoints are derived from the same base Qwen2.5-VL-7B and share compatible, aligned parameterization.
- domain assumption The fixed development partitions are representative of the test distributions for all six benchmarks.
- ad hoc to paper FFN neurons with high activation magnitude and output contribution on expert behavior spans are the causal carriers of behavior-critical actions.
- domain assumption The DGC workflow using gpt-5.5 correctly identifies behavior-critical gaps and protection behaviors from trajectories.
Cite this review
Pith. "Pith review of AgentPatch: Coarse-to-Fine Weak-Task Repair for Merging Agentic Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/5HTKPNM2
@misc{pith2026260806699,
author = {Pith},
title = {Pith review of: AgentPatch: Coarse-to-Fine Weak-Task Repair for Merging Agentic Multimodal Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/5HTKPNM2}},
note = {Machine review of arXiv:2608.06699}
}
read the original abstract
Agentic multimodal large language models (MLLMs) extend multimodal perception and reasoning with planning, tool use, and interaction in dynamic environments. Yet current models are specialized for particular tools or environments, complicating consolidation into a single generalist. We formulate Agentic MLLM Merging and identify two challenges: asymmetric capability preservation, whereby capabilities with different interaction complexity are retained unevenly, producing weak tasks after merging, and behavior-critical forgetting, whereby losing decisive actions can derail long-horizon execution. We propose AgentPatch, a training-free coarse-to-fine repair framework. It selects a stable merged backbone, restores diluted weak-task-specific signals through Weak-Task Unique Residual Recovery, and applies an Agent-Guided Behavior-Critical Patch that recovers decisive behaviors under explicit capability protection. AgentPatch produces a single static checkpoint without routing or ensembles. Experiments across six agentic and multimodal benchmarks show that AgentPatch improves diverse merged backbones, alleviates weak-task degradation, and better balances weak-task recovery with the preservation of complementary search and agentic visual processing capabilities. Code is available at https://github.com/ziboshao/AgentPatch.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Qwen3-VL Technical Report.arXiv preprint arXiv:2511.21631
Bai, S.; Cai, Y.; Chen, R.; Chen, K.; Chen, X.; Cheng, Z.; Deng,L.;Ding,W.;Gao,C.;Ge,C.;etal.2025a. Qwen3-VL Technical Report.arXiv preprint arXiv:2511.21631. Bai,S.;Chen,K.;Liu,X.;Wang,J.;Ge,W.;Song,S.;Dang, K.; Wang, P.; Wang, S.; Tang, J.; Zhong, H.; Zhu, Y.; Yang, M.; Li, Z.; Wan, J.; Wang, P.; Ding, W.; Fu, Z.; Xu, Y.; Ye, J.; Zhang, X.; Xie, T.; Che...
-
[3]
Compiler selector template. [SYSTEM PROMPT] You are the Selector Compiler for expert-directed behavior-gap repair. Combine the Diagnoser behavior-gap groups and Guardian protection groups into the executable selector specification consumed by activation calibration. Preserve valid matcher fields and convert matcher hints into literal, regular-expression, ...
work page 1915
-
[6]
Web- watcher: Breaking new frontier of vision-language deep re- search agent.arXiv preprint arXiv:2508.05748. Ilharco, G.; Ribeiro, M. T.; Wortsman, M.; Gururangan, S.; Schmidt, L.; Hajishirzi, H.; and Farhadi, A
-
[7]
Jang,D.-H.;Yun,S.;andHan,D.2024
Editing models with task arithmetic.arXiv preprint arXiv:2212.04089. Jang,D.-H.;Yun,S.;andHan,D.2024. Modelstock:Allwe needisjustafewfine-tunedmodels.InEuropean Conference on Computer Vision, 207–223. Springer. Jiang, D.; Zhang, R.; Guo, Z.; Wu, Y.; Lei, J.; Qiu, P.; Lu, P.; Chen, Z.; Fu, C.; Song, G.; Gao, P.; Liu, Y.; Li, C.; and Li, H
arXiv 2024
-
[8]
Li,F.;Zhang,R.;Zhang,H.;Zhang,Y.;Li,B.;Li,W.;Ma,Z.; and Li, C
MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines.arXiv preprint arXiv:2409.12959. Li,F.;Zhang,R.;Zhang,H.;Zhang,Y.;Li,B.;Li,W.;Ma,Z.; and Li, C
-
[9]
Li, J.; Li, D.; Savarese, S.; and Hoi, S
LLaVA-NeXT-Interleave: Tackling Multi- Image, Video, and 3D in Large Multimodal Models.arXiv preprint arXiv:2407.07895. Li, J.; Li, D.; Savarese, S.; and Hoi, S
-
[11]
Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326. Rawles, C.; Clinckemaillie, S.; Chang, Y.; Waltz, J.; Lau, G.; Fair, M.; Li, A.; Bishop, W.; Li, W.; Campbell-Ajala, F.; Toyama, D.; Berry, R.; Tyamagundlu, D.; Lillicrap, T.; and Riva, O
-
[12]
Shao, Z.; Xiong, B.; Yang, X.; Song, Y.; Zhang, Q.; Chen, H.; and Xu, C
AndroidWorld: A Dynamic Benchmark- ing Environment for Autonomous Agents.arXiv preprint arXiv:2405.14573. Shao, Z.; Xiong, B.; Yang, X.; Song, Y.; Zhang, Q.; Chen, H.; and Xu, C
Show all 25 references
-
[13]
Stoica, G.; Ramesh, P.; Ecsedi, B.; Choshen, L.; and Hoff- man, J
PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merg- ing.arXiv preprint arXiv:2604.22823. Stoica, G.; Ramesh, P.; Ecsedi, B.; Choshen, L.; and Hoff- man, J
-
[14]
arXiv preprint arXiv:2410.19735
Model Merging with SVD to Tie the Knots. arXiv preprint arXiv:2410.19735. Sung, Y.-L.; Li, L.; Lin, K.; Gan, Z.; Bansal, M.; and Wang, L
-
[15]
arXiv preprint arXiv:2304.14933
An empirical study of multimodal model merging. arXiv preprint arXiv:2304.14933. Wang, W.; Ding, L.; Zeng, M.; Zhou, X.; Shen, L.; Luo, Y.; and Tao, D
-
[16]
Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Per- ception in Multimodal Large Language Models.arXiv preprint arXiv:2408.15556. Wang,W.;Gao,Z.;Gu,L.;Pu,H.;Cui,L.;Wei,X.;Liu,Z.; Jing,L.;Ye,S.;Shao,J.;etal.2025.InternVL3.5:Advancing Open-Source...
2025 arXiv
-
[17]
Wortsman, M.; Ilharco, G.; Gadre, S
Unifying Multi- modal Large Language Model Capabilities and Modalities via Model Merging.arXiv preprint arXiv:2505.19892. Wortsman, M.; Ilharco, G.; Gadre, S. Y.; Roelofs, R.; Gontijo-Lopes, R.; Morcos, A. S.; Namkoong, H.; Farhadi, A.;Carmon,Y.;Kornblith,S.;etal.2022. ModelSo...
2022
-
[18]
arXiv preprint arXiv:2506.20670
Mmsearch-r1: Incentivizing lmms to search. arXiv preprint arXiv:2506.20670. Wu, P.; and Xie, S
-
[19]
Xie, T.; Zhang, D.; Chen, J.; Li, X.; Zhao, S.; Cao, R.; Toh, J
V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs.arXiv preprint arXiv:2312.14135. Xie, T.; Zhang, D.; Chen, J.; Li, X.; Zhao, S.; Cao, R.; Toh, J. H.; Cheng, Z.; Shin, D.; Lei, F.; Liu, Y.; Xu, Y.; Zhou, S.; Savarese, S.; Xiong, C.; Zhong, V.; and Yu, T
-
[20]
Xu, B.; Wu, H.; Lin, H.; Huang, W.; Zhu, B.; Shu, Y.; and Qin, C
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments.arXiv preprint arXiv:2404.07972. Xu, B.; Wu, H.; Lin, H.; Huang, W.; Zhu, B.; Shu, Y.; and Qin, C. 2026a. ACE-Merging: Data-Free Model Merg- ing with Adaptive Covariance Estimation.arXiv...
-
[21]
Yao, H.; Zhang, R.; Huang, J.; Zhang, J.; Wang, Y.; Fang, B.;Zhu,R.;Jing,Y.;Liu,S.;Li,G.;etal.2025
Ties-merging: Resolving interference when merg- ing models.Advances in Neural Information Processing Systems, 36: 7093–7115. Yao, H.; Zhang, R.; Huang, J.; Zhang, J.; Wang, Y.; Fang, B.;Zhu,R.;Jing,Y.;Liu,S.;Li,G.;etal.2025. Asurveyon agentic multimodal large language models.a...
2025
-
[22]
Yu,L.;Yu,B.;Yu,H.;Huang,F.;andLi,Y.2024
Mobile-agent- v3: Fundamental agents for gui automation.arXiv preprint arXiv:2508.15144. Yu,L.;Yu,B.;Yu,H.;Huang,F.;andLi,Y.2024. Language Models Are Super Mario: Absorbing Abilities from Homol- ogous Models as a Free Lunch. InInternational Conference on Machine Learning. Zeng...
2024 arXiv
-
[23]
InNeurIPS 2025 Fourth Workshop on Deep Learning for Code
Thyme: Think beyondimages. InNeurIPS 2025 Fourth Workshop on Deep Learning for Code. Zheng, Z.; Yang, M.; Hong, J.; Zhao, C.; Xu, G.; Yang, L.; Shen, C.; and Yu, X
2025
-
[24]
thinking with images
Deepeyes: Incentivizing" thinking with images" via reinforcement learning.arXiv preprint arXiv:2505.14362. Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023.MiniGPT-4:EnhancingVision-LanguageUnderstand- ing with Advanced Large Language Models.arXiv preprint arXiv:230...
2023 arXiv
-
[2022]
Dai, W.; Li, J.; Li, D.; Tiong, A.; Zhao, J.; Wang, W.; Li, B.; Fung, P
Fusing Finetuned Models for Better Pretraining.arXiv preprint arXiv:2204.03044. Dai, W.; Li, J.; Li, D.; Tiong, A.; Zhao, J.; Wang, W.; Li, B.; Fung, P. N.; and Hoi, S
-
[2023]
BLIP-2: Boot- strapping Language-Image Pre-Training with Frozen Image EncodersandLargeLanguageModels.InInternational Con- ference on Machine Learning, 19730–19742. PMLR. Liu,H.;Li,C.;Wu,Q.;andLee,Y.J.2023.VisualInstruction Tuning.Advances in Neural Information Processing Syste...
2023
-
[2024]
Choshen, L.; Venezian, E.; Slonim, N.; and Katz, Y
Model composi- tion for multimodal large language models.arXiv preprint arXiv:2402.12750. Choshen, L.; Venezian, E.; Slonim, N.; and Katz, Y
-
[2025]
In Proceedings of the Computer Vision and Pattern Recogni- tion Conference, 18695–18705
Task singular vectors: Reducing task interference in model merging. In Proceedings of the Computer Vision and Pattern Recogni- tion Conference, 18695–18705. Garipov,T.;Izmailov,P.;Podoprikhin,D.;Vetrov,D.P.;and Wilson,A.G.2018. LossSurfaces,ModeConnectivity,and Fast Ensembling...
2018
-
[2026]
Gargiulo, A
ARM: Role-Conditioned Neuron Transplantation for Training- Free Generalist LLM Agent Merging.arXiv preprint arXiv:2601.07309. Gargiulo, A. A.; Crisostomi, D.; Bucarelli, M. S.; Scarda- pane, S.; Silvestri, F.; and Rodola, E
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.