Pith. sign in

REVIEW 2 major objections 44 references

Merging coefficients for biological multimodal language models should come from embedding-space signals, not parameter-space heuristics, to capture modality specialization and improve both cross-modal reasoning and single-modal knowledge.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 21:17 UTC pith:YAECKVPU

load-bearing objection We cannot evaluate ES-Merging: the supplied full manuscript is a different paper (OCRA, robotics), so the merging claims have no body to audit. the 2 major comments →

arxiv 2603.14405 v2 pith:YAECKVPU submitted 2026-03-15 cs.LG cs.AI

ES-Merging: Biological MLLM Merging via Embedding Space Signals

Wonbin Lee , Dongki Kim , Sung Ju Hwang This is my paper
classification cs.LG cs.AI
keywords model mergingbiological MLLMsmultimodal large language modelsembedding space signalsmodality specializationlayer-wise coefficientselement-wise coefficients
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Biological multimodal large language models are powerful for science but are typically specialized to one modality, which blocks inherently cross-modal problems. Model merging can fuse them into a single model without full retraining, yet existing methods rely on input-agnostic rules in parameter space that do not faithfully reflect how each modality specializes. This paper introduces ES-Merging, which estimates the merging coefficients from signals living in embedding space instead. Coarse-grained embedding signals set layer-wise coefficients and fine-grained signals set element-wise coefficients; the two are combined so they complement each other. Experiments show the resulting merged models outperform prior merging methods on cross-modal reasoning while also better preserving single-modal knowledge, supporting the claim that embedding signals are a more principled basis for this kind of fusion.

Core claim

ES-Merging estimates merging coefficients from embedding-space signals rather than parameter-space heuristics: coarse-grained signals produce layer-wise coefficients, fine-grained signals produce element-wise coefficients, and the two are jointly combined. The resulting merged biological MLLMs outperform existing merging methods on both cross-modal reasoning and single-modal knowledge preservation.

What carries the argument

ES-Merging: a framework that moves coefficient estimation from input-agnostic parameter-space heuristics to embedding-space signals, using coarse-grained signals for layer-wise coefficients and fine-grained signals for element-wise coefficients that are jointly combined to capture modality specialization.

Load-bearing premise

Embedding-space signals capture modality specialization more faithfully than parameter-space heuristics, and the layer-wise plus element-wise coefficients estimated from them are complementary rather than redundant or noisy.

What would settle it

On a fixed suite of biological cross-modal reasoning and single-modal retention benchmarks, if an ES-Merging model systematically underperforms strong parameter-space merging baselines on both metrics, the claim that embedding signals supply a better foundation for the coefficients would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Specialized single-modality biological models can be fused into unified MLLMs without expensive joint retraining.
  • Cross-modal scientific tasks become solvable by the merged model while single-modal performance is better retained.
  • Embedding-space diagnostics become a practical tool for measuring and controlling modality specialization inside large models.
  • Subsequent merging recipes for other multimodal foundation models can start from embedding signals rather than weight-space heuristics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same coarse-plus-fine embedding-signal recipe may transfer to non-biological multimodal merges such as vision-language or audio-language systems.
  • If embedding signals encode specialization cleanly, they could also guide selective unmerging or modular editing of individual modalities after fusion.
  • Coefficient estimation from embeddings may reduce reliance on large held-out validation sets that many current merging methods require.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The manuscript claims to introduce ES-Merging, a framework that estimates layer-wise and element-wise merging coefficients for biological multimodal large language models from coarse- and fine-grained embedding-space signals rather than input-agnostic parameter-space heuristics, and that this yields superior cross-modal reasoning while preserving single-modal knowledge. The supplied full text, however, is an entirely different paper (OCRA) on object-centric human-to-robot action transfer that uses multi-view RGB, VGGT reconstruction, tactile pretraining, ResFiLM fusion, and a diffusion policy. No equations, coefficient estimators, biological MLLM experiments, or merging results for ES-Merging appear in the body.

Significance. If the abstract claims were supported by a matching technical body, the shift from parameter-space to embedding-space signals for MLLM merging would be a useful contribution to efficient multimodal foundation-model construction in the life sciences. As submitted, the mismatch between title/abstract and full text means no such contribution can be evaluated or credited. The OCRA material that is present is a coherent robotics system paper, but it does not address the stated ES-Merging claims.

major comments (2)
  1. Title, abstract, and arXiv identifier assert ES-Merging for biological MLLM merging via embedding-space signals, yet the entire body (Sections I–VI, Figs. 1–6, Tables I–V, and all equations) describes OCRA, a robotics pipeline for human-to-robot transfer. There are no layer-wise/element-wise coefficient estimators, no embedding-signal definitions, no biological modalities, and no merging experiments. The central claim is therefore unsupported by any method, result, or analysis in the manuscript.
  2. Because the body contains none of the promised ES-Merging content, the abstract’s assertions of outperformance on cross-modal reasoning and single-modal knowledge preservation, and of complementary coarse-/fine-grained signals, cannot be audited. No baselines, ablations, datasets, or statistical tests for MLLM merging exist in the provided text.

Circularity Check

0 steps flagged

No circular derivation found: claimed ES-Merging body is absent (supplied text is unrelated OCRA robotics paper); OCRA itself is empirical, not self-definitional.

full rationale

The abstract asserts ES-Merging estimates layer-wise and element-wise merging coefficients from coarse- and fine-grained embedding-space signals and that this outperforms parameter-space heuristics. No equations, coefficient definitions, fitting procedures, or embedding analyses for that claim appear in the supplied full text, which is instead OCRA (arXiv:2603.14401)—a human-to-robot imitation system using multi-view VGGT reconstruction, object-centric masks, tactile MAE pretraining, ResFiLM fusion, and a diffusion policy, evaluated by real-world success rates and ablations. Circularity patterns (self-definitional X↔Y, fitted inputs renamed as predictions, load-bearing self-citation uniqueness, ansatz smuggled via self-citation, renaming known results) require a walkable derivation chain with quoted reductions; none exists for ES-Merging here, and OCRA’s pipeline is design-plus-empirical-benchmark rather than a closed algebraic or fitted tautology. Self-citations in OCRA related work are ordinary background, not uniqueness theorems forcing the result. Honest non-finding: score 0, no circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 1 invented entities

With only the abstract, the ledger is thin. The central claim rests on domain assumptions that specialist biological MLLMs exist and are mergeable, that embedding signals reflect modality specialization better than parameter heuristics, and that joint layer-wise plus element-wise coefficients are complementary. No free parameters or invented physical entities are named in the abstract; any fitted merge scales or embedding statistics would appear in the missing method section.

axioms (3)
  • domain assumption Existing biological MLLMs are specialized to a single modality and therefore limited on inherently cross-modal scientific problems.
    Stated as the opening problem setup in the abstract; load-bearing for why merging is needed.
  • domain assumption Input-agnostic parameter-space merging heuristics fail to faithfully capture modality specialization.
    Abstract’s stated limitation of prior work; motivates moving to embedding signals.
  • ad hoc to paper Coarse-grained and fine-grained embedding-space signals can be used to estimate complementary layer-wise and element-wise merging coefficients.
    Core methodological premise of ES-Merging as described in the abstract; not independently established in the available text.
invented entities (1)
  • ES-Merging (Embedding-Signal-based MLLM Merging framework) no independent evidence
    purpose: Estimate and apply layer-wise and element-wise merge coefficients from embedding signals to unify single-modality biological MLLMs.
    Named framework introduced by the paper; independent evidence of superiority is claimed via experiments not present in the available text.

pith-pipeline@v1.1.0-grok45 · 17463 in / 2592 out tokens · 25411 ms · 2026-07-14T21:17:13.080463+00:00 · methodology

0 comments
read the original abstract

Biological multimodal large language models (MLLMs) have emerged as powerful foundation models for scientific discovery. However, existing models are specialized to a single modality, limiting their ability to solve inherently cross-modal scientific problems. While model merging is an efficient method to combine the different modalities into a unified MLLM, existing methods rely on input-agnostic parameter space heuristics that fail to faithfully capture modality specialization. To overcome this limitation, we propose the Embedding-Signal-based MLLM Merging (ES-Merging), a framework that estimates merging coefficients from embedding space signals, moving the merging paradigm from the parameter signals to the embedding signals. ES-Merging exploits coarse-grained and fine-grained signals from embedding space to estimate the layer-wise and element-wise merging coefficients, respectively, which are jointly combined for complementary coefficient estimation. Through extensive experiments, we demonstrate that ES-Merging outperforms existing merging methods not only on the cross-modal reasoning but also on the single-modal knowledge preservation, establishing that embedding space signals provide a principled and effective foundation for MLLM merging.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 15 linked inside Pith

  1. [1]

    Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots,

    C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song, “Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots,”arXiv preprint arXiv:2402.10329, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Fastumi: A scalable and hardware- independent universal manipulation interface with dataset,

    Z. Zhaxizhuoma, K. Liu, C. Guan, Z. Jia, Z. Wu, X. Liu, T. Wang, S. Liang, P. Chen, P. Zhanget al., “Fastumi: A scalable and hardware- independent universal manipulation interface with dataset,” inConfer- ence on Robot Learning. PMLR, 2025, pp. 3069–3093

    2025

  3. [3]

    R3m: A universal visual representation for robot manipulation,

    S. Nair, A. Rajeswaran, V . Kumar, C. Finn, and A. Gupta, “R3m: A universal visual representation for robot manipulation,”arXiv preprint arXiv:2203.12601, 2022

    Pith/arXiv arXiv 2022

  4. [4]

    Vip: Towards universal visual reward and representa- tion via value-implicit pre-training,

    Y . J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V . Kumar, and A. Zhang, “Vip: Towards universal visual reward and representa- tion via value-implicit pre-training,”arXiv preprint arXiv:2210.00030, 2022

    Pith/arXiv arXiv 2022

  5. [5]

    Spot: Se (3) pose trajectory diffusion for object-centric manipulation,

    C.-C. Hsu, B. Wen, J. Xu, Y . Narang, X. Wang, Y . Zhu, J. Biswas, and S. Birchfield, “Spot: Se (3) pose trajectory diffusion for object-centric manipulation,” in2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 4853–4860

    2025

  6. [6]

    Learning generalizable manipulation policies with object-centric 3d representations,

    Y . Zhu, Z. Jiang, P. Stone, and Y . Zhu, “Learning generalizable manipulation policies with object-centric 3d representations,”arXiv preprint arXiv:2310.14386, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    Vggt: Visual geometry grounded transformer,

    J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny, “Vggt: Visual geometry grounded transformer,” inPro- ceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 5294–5306

    2025

  8. [8]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection,

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Suet al., “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,” inECCV, 2024

    2024

  9. [9]

    Sam 2: Segment anything in images and videos,

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafsonet al., “Sam 2: Segment anything in images and videos,”arXiv preprint arXiv:2408.00714, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Diffusion policy: Visuomotor policy learning via action diffusion,

    C. Chi, S. Feng, Y . Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,”The International Journal of Robotics Research, vol. 44, pp. 1684 – 1704, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:257378658

    2023

  11. [11]

    Learning fine-grained bimanual manipulation with low-cost hardware,

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,”arXiv preprint arXiv:2304.13705, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    π 0: A vision- language-action flow model for general robot control,

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichteret al., “π 0: A vision- language-action flow model for general robot control,”arXiv preprint arXiv:2410.24164, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Open- vla: An open-source vision-language-action model,

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketiet al., “Open- vla: An open-source vision-language-action model,”arXiv preprint arXiv:2406.09246, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Alvinn: An autonomous land vehicle in a neural network,

    D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,”NeurIPS, 1988

    1988

  15. [15]

    Ego4d: Around the world in 3,000 hours of egocentric video,

    K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liuet al., “Ego4d: Around the world in 3,000 hours of egocentric video,” inCVPR, 2022

    2022

  16. [16]

    Foundationpose: Unified 6d pose estimation and tracking of novel objects,

    B. Wen, W. Yang, J. Kautz, and S. T. Birchfield, “Foundationpose: Unified 6d pose estimation and tracking of novel objects,”CVPR, 2024

    2024

  17. [17]

    Compositional scene represen- tation learning via reconstruction: A survey,

    J. Yuan, T. Chen, B. Li, and X. Xue, “Compositional scene represen- tation learning via reconstruction: A survey,”IEEE TPAMI, 2023

    2023

  18. [18]

    Scalor: Generative world models with scalable object representations,

    J. Jiang, S. Janghorbani, G. de Melo, and S. Ahn, “Scalor: Generative world models with scalable object representations,” inICLR, 2020

    2020

  19. [19]

    Space: Unsupervised object-oriented scene representation via spatial attention and decomposition,

    Z. Lin, Y .-F. Wu, S. V . Peri, W. Sun, G. Singh, F. Deng, J. Jiang, and S. Ahn, “Space: Unsupervised object-oriented scene representation via spatial attention and decomposition,” inICLR, 2020

    2020

  20. [20]

    Object-centric learning with slot attention,

    F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf, “Object-centric learning with slot attention,”NeurIPS, 2020

    2020

  21. [21]

    Bridging the gap to real-world object-centric learning,

    M. Seitzer, M. Horn, A. Zadaianchuk, D. Zietlow, T. Xiao, C.- J. Simon-Gabriel, T. He, Z. Zhang, B. Sch ¨olkopf, T. Broxet al., “Bridging the gap to real-world object-centric learning,”ICLR, 2023

    2023

  22. [22]

    Conditional object- centric learning from video,

    T. Kipf, G. F. Elsayed, A. Mahendran, A. Stone, S. Sabour, G. Heigold, R. Jonschkowski, A. Dosovitskiy, and K. Greff, “Conditional object- centric learning from video,”arXiv preprint arXiv:2111.12594, 2021

    Pith/arXiv arXiv 2021

  23. [23]

    Simple unsupervised object-centric learning for complex and naturalistic videos,

    G. Singh, Y .-F. Wu, and S. Ahn, “Simple unsupervised object-centric learning for complex and naturalistic videos,”NeurIPS, 2022

    2022

  24. [24]

    Self-supervised vi- sual reinforcement learning with object-centric representations,

    A. Zadaianchuk, M. Seitzer, and G. Martius, “Self-supervised vi- sual reinforcement learning with object-centric representations,”arXiv preprint arXiv:2011.14381, 2020

    Pith/arXiv arXiv 2011

  25. [25]

    An investigation into pre-training object-centric representations for reinforcement learning,

    J. Yoon, Y .-F. Wu, H. Bae, and S. Ahn, “An investigation into pre-training object-centric representations for reinforcement learning,” arXiv preprint arXiv:2302.04419, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    Focus: object- centric world models for robotic manipulation,

    S. Ferraro, P. Mazzaglia, T. Verbelen, and B. Dhoedt, “Focus: object- centric world models for robotic manipulation,”Frontiers in Neuro- robotics, vol. 19, p. 1585386, 2025

    2025

  27. [27]

    Sold: Reinforcement learning with slot object-centric latent dynamics,

    M. Mosbach, J. N. Ewertz, A. Villar-Corrales, and S. Behnke, “Sold: Reinforcement learning with slot object-centric latent dynamics,” arXiv preprint arXiv:2410.08822, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Deep object-centric representations for generalizable robot learning,

    C. Devin, P. Abbeel, T. Darrell, and S. Levine, “Deep object-centric representations for generalizable robot learning,” inICRA, 2018

    2018

  29. [29]

    Deep object- centric policies for autonomous driving,

    D. Wang, C. Devin, Q.-Z. Cai, F. Yu, and T. Darrell, “Deep object- centric policies for autonomous driving,” inICRA, 2019

    2019

  30. [30]

    Deep object pose estimation for semantic robotic grasping of household objects,

    J. Tremblay, T. To, B. Sundaralingam, Y . Xiang, D. Fox, and S. Birch- field, “Deep object pose estimation for semantic robotic grasping of household objects,”arXiv preprint arXiv:1809.10790, 2018

    Pith/arXiv arXiv 2018

  31. [31]

    Self-supervised 6d object pose estimation for robot manipulation,

    X. Deng, Y . Xiang, A. Mousavian, C. Eppner, T. Bretl, and D. Fox, “Self-supervised 6d object pose estimation for robot manipulation,” in ICRA, 2020

    2020

  32. [32]

    Viola: Imitation learning for vision-based manipulation with object proposal priors,

    Y . Zhu, A. Joshi, P. Stone, and Y . Zhu, “Viola: Imitation learning for vision-based manipulation with object proposal priors,” inCoRL, 2023

    2023

  33. [33]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Loet al., “Segment anything,” inICCV, 2023

    2023

  34. [34]

    Emerging properties in self-supervised vision trans- formers,

    M. Caron, H. Touvron, I. Misra, H. J ´egou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision trans- formers,” inICCV, 2021

    2021

  35. [35]

    Dinov2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khali- dov, P. Fernandez, D. Haziza, F. Massa, A. El-Noubyet al., “Dinov2: Learning robust visual features without supervision,”arXiv preprint arXiv:2304.07193, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    Gelsight: High-resolution robot tactile sensors for estimating geometry and force,

    W. Yuan, S. Dong, and E. H. Adelson, “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,”Sensors, 2017

    2017

  37. [37]

    Gelslim: A high-resolution, compact, robust, and calibrated tactile- sensing finger,

    E. Donlon, S. Dong, M. Liu, J. Li, E. Adelson, and A. Rodriguez, “Gelslim: A high-resolution, compact, robust, and calibrated tactile- sensing finger,” inIROS, 2018

    2018

  38. [38]

    3d-vitac: Learning fine-grained manipulation with visuo-tactile sensing,

    B. Huang, Y . Wang, X. Yang, Y . Luo, and Y . Li, “3d-vitac: Learning fine-grained manipulation with visuo-tactile sensing,” inCoRL,

  39. [39]

    Available: https://api.semanticscholar.org/CorpusID: 273707578

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 273707578

  40. [40]

    Freetac- man: Robot-free visuo-tactile data collection system for contact-rich manipulation,

    L. Wu, C. Yu, J. Ren, L. Chen, R. Huang, G. Gu, and H. Li, “Freetac- man: Robot-free visuo-tactile data collection system for contact-rich manipulation,”arXiv preprint arXiv:2506.01941, 2025

    arXiv 2025

  41. [41]

    A novel flexible multidimensional force platform for assessing regional plantar pressure and shear stress under the foot during gait,

    H. Luo, X. Zhang, Y . Xu, J. Yu, X. Ma, and W.-M. Chen, “A novel flexible multidimensional force platform for assessing regional plantar pressure and shear stress under the foot during gait,”IEEE Sensors Journal, 2024

    2024

  42. [42]

    3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations,

    Y . Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu, “3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations,”arXiv preprint arXiv:2403.03954, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Masked autoencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Doll’ar, and R. B. Girshick, “Masked autoencoders are scalable vision learners,”CVPR, 2022

    2022

  44. [44]

    Robotic compliant object prying using diffusion policy guided by vision and force observations,

    J. H. Kang, S. Joshi, R. Huang, and S. K. Gupta, “Robotic compliant object prying using diffusion policy guided by vision and force observations,”IEEE RA-L, 2025

    2025