Pith. sign in

REVIEW 4 major objections 6 minor 56 references

A Bloom-inspired hierarchy turns multi-expert vision outputs into staged, citeable reasoning that LVLMs can use at inference time.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 09:11 UTC pith:Z7FQWQNX

load-bearing objection Clean training-free recipe that makes multi-expert LVLM evidence citeable and traces Bloom-style steps, with real but modest gains that mostly ride the expert pool rather than the hierarchy itself. the 4 major comments →

arxiv 2607.10796 v1 pith:Z7FQWQNX submitted 2026-07-12 cs.CV

Mixture of Cognitive Experts in Large Vision-Language Models

Robert Wijaya , Ngai-Man Cheung This is my paper
classification cs.CV
keywords vision-language modelsmixture of expertsBloom taxonomycognitive verbalizationevidence groundinghallucination reductionreasoning tracemultimodal reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large vision-language models still struggle to ground answers in fine visual detail and often apply the same reasoning path to every query. This paper claims that specialized computer-vision models already supply the missing perceptual pieces—object detection, masks, OCR, charts—and that the missing architectural step is an explicit, hierarchical protocol for turning those outputs into evidence and then into staged reasoning. The proposed system first decomposes expert outputs into short, non-inferential statements stored with type and grounding references. It next runs a Bloom-inspired verbalizer that starts at a query-determined cognitive level and walks through list–explain–illustrate–compare–hypothesize–final, citing evidence IDs at each step. A lightweight parser recovers the start level, executed steps, and evidence-use statistics. The authors show that this training-free pipeline improves scores on perception- and reasoning-heavy benchmarks and that different questions really do enter the hierarchy at different levels, making the model’s reasoning trajectory measurable.

Core claim

Imposing an evidence-grounded, hierarchical (Bloom-inspired) reasoning protocol on multi-expert LVLM inference produces more verifiable and analyzable multimodal behavior and yields measurable gains on perception- and reasoning-intensive real-world scene-understanding benchmarks, all without additional training.

What carries the argument

Two-stage cognitive verbalization: a Literal Evidence Summary that atomizes expert outputs into typed, citeable statements, followed by Bloom Verbalization that converts those statements into an ordered, evidence-cited reasoning trace consumed by the final LLM; a Reasoning Trace Module then recovers start level, step set, and evidence-use statistics.

Load-bearing premise

That the hand-designed Bloom step order and template-based atomic verbalizers produce faithfully non-inferential evidence that a frozen base language model can reliably use at inference time without any further training.

What would settle it

On a held-out suite of perception- and reasoning-intensive VQA items, remove or scramble the Bloom stages while keeping the same expert pool and measure whether the accuracy gains and the measured start-level/evidence-citation statistics disappear.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MoCE, a training-free LVLM inference framework that routes specialized vision experts (detection, segmentation, OCR/chart parsers), converts their outputs into a typed, citeable Literal Evidence Summary of atomic statements, then applies a Bloom-inspired staged verbalizer (LIST→EXPLAIN→ILLUSTRATE→COMPARE→HYPOTHESIZE→FINAL) to produce an evidence-cited draft that is fed with the image and query to a LLaVA-style Vicuna-7B backbone. A lightweight Reasoning Trace Module parses the draft to recover query-conditioned entry levels, step-wise citations, and evidence-type usage. Empirically, MoCE reports gains over InstructBLIP, Qwen-VL, and LLaVA-1.5/NeXT on TextVQA, MMBench, MMVet, POPE, and SEED, with ability breakdowns favoring perception and relational reasoning; an experts-only vs. experts+Bloom ablation and N=5000 trace diagnostics are used to argue that the hierarchical protocol improves verifiability and performance.

Significance. If the hierarchical, evidence-grounded protocol is shown to be the operative ingredient—not merely the expert pool, longer intermediate context, or adapter fusion—the work would offer a practical, training-free way to make multi-expert LVLM reasoning more inspectable and less hallucination-prone, with a reusable diagnostic (the Reasoning Trace Module) that quantifies cognitive entry levels and evidence use. The explicit typed evidence store with citation IDs and the deterministic trace parser are concrete engineering contributions that go beyond typical CoT prompting. Significance is currently limited by incomplete isolation of the Bloom stage from the MoVA-style expert stack the method reuses, so the architectural claim is promising but not yet firmly established as a cognitive-protocol result rather than a multi-expert system result.

major comments (4)
  1. §4.3, Table 2 is the load-bearing isolation of Bloom synthesis from the expert pool, yet it is incomplete for the central claim. Experts Only already reaches MMB 70.4 / MM-Vet 49.3 / SEED-Text 47.06; adding Bloom lifts MMB (+5.3) and SEED-Text (+4.7) but leaves MM-Vet flat-to-worse (49.3→49.0) while increasing latency (10.24s→13.40s), GFLOPs (+13.75%), and average tokens (63.70→76.87). There is no control that appends unstructured intermediate text or free-form CoT of matched length/token budget, so gains cannot yet be attributed to the ordered Bloom hierarchy rather than extra context or more verbose generation. A length-matched non-hierarchical scaffold is needed before concluding that the protocol itself improves verifiable multimodal behavior.
  2. §4.2 Table 1 and Fig. 4 compare MoCE only to non-expert LVLMs (InstructBLIP, Qwen-VL, LLaVA-1.5/NeXT). The method reuses the expert pool and adapter design from MoVA (§3.1, §3.5, ref. [55]), and Vision Experts Only is already a strong intermediate system in Table 2. Without a main-table comparison to MoVA (or an equivalently expert-augmented baseline under the same backbone and inference budget), the performance narrative over-attributes gains to the cognitive verbalization framework relative to prior multi-encoder expert integration. Reporting MoVA / Experts Only alongside MoCE in Table 1 and Fig. 4 is necessary for the contribution claim.
  3. §4.3 Table 3 reports step-present rates of only ~14–16% per Bloom step over N=5000, while among N=952 detectable traces 83.82% start at LIST. These figures undercut the claim that the hierarchy is systematically operationalized at inference time (§3.2, conclusion). The manuscript should clarify the relationship between “step present,” “detected traces,” and hierarchy compliance (skips, out-of-order headers, missing FINAL), quantify how often y_Bloom is malformed or empty of structure, and analyze whether performance gains concentrate on the minority of examples that actually execute multi-step traces. Without that, the Reasoning Trace Module’s diagnostics do not yet support “query-conditioned cognitive trajectories” as a reliable property of the system.
  4. §3.2–3.4 leave the implementation of f_Bloom and the router under-specified for a training-free, reproducibility-critical pipeline. It is unclear whether Bloom verbalization and expert selection are produced by the same Vicuna-7B via fixed prompts, a separate LLM, or templates; how ℓ0 is chosen from q; and what the exact routing prompt and top-K are. Because the paper’s free parameters include router top-K, adapter depth L, and entry-level logic, and because atomic verbalization is claimed to be non-inferential, these details (and sensitivity to them) are load-bearing for the training-free and fidelity claims in §4.1.
minor comments (6)
  1. Several typographical and formatting issues: “V erbalization” / “T race” with stray spaces in headings; “strenghten” (Related Works); “L VLMs”; “Course Perception” in Fig. 4 should be “Coarse Perception.”
  2. §3.2 Eq. (4)–(5): notation for ˜T and the Reasoning Structure/Trace Module is slightly inconsistent across text and figure captions; unify naming.
  3. Fig. 1 and Fig. 3 are helpful but do not show failure cases (wrong expert routing, hallucinated citations, hierarchy skips); one negative qualitative example would strengthen the interpretability narrative.
  4. Related work positions MoAI/MoVA well but could more sharply state what is new relative to those systems beyond the Bloom scaffold (e.g., typed evidence store + deterministic trace metrics).
  5. Appendix tables on TEXT evidence atomicity are useful; a brief pointer from §3.2 would help readers find them.
  6. All benchmark numbers are point estimates with no seeds, variance, or significance tests; even for a training-free method, multi-run or bootstrap intervals on the main deltas would improve credibility.

Circularity Check

0 steps flagged

No circularity: training-free architectural scaffold evaluated on external public benchmarks; Bloom hierarchy is an imposed protocol, not a prediction forced by its inputs.

full rationale

This is an empirical systems paper, not a first-principles derivation. The load-bearing claims are (i) that routing specialized CV experts into a Literal Evidence Summary plus Bloom-structured draft improves LVLM perception/reasoning, and (ii) that a deterministic Reasoning Trace Module makes evidence use and entry levels measurable. Both are tested against external public suites (TextVQA, MMBench, MMVet, POPE, SEED) under a stated training-free regime (§4.1); no parameter is fitted to a subset of those scores and then re-reported as a prediction. The Bloom step order L = [LIST, EXPLAIN, ILLUSTRATE, COMPARE, HYPOTHESIZE, FINAL ANS.] is an explicit design choice inspired by Bloom [10], not a quantity defined in terms of the reported accuracies. Expert pool and adapter are taken from prior external work (MoVA [55], DETR, SAM, Pix2Struct, etc.), not from an unverified self-citation uniqueness theorem. The authors’ own prior citation [50] is peripheral and not load-bearing for the central results. Weak isolation of Bloom vs. experts in Table 2 is a methodological/attribution concern, not circular reduction of a claimed prediction to its inputs. Therefore the derivation chain does not collapse by construction; score 0 with no circular steps.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 3 invented entities

The central empirical claim rests on three domain assumptions (Bloom hierarchy suitability, literality of expert verbalization, router competence) plus a handful of hand-chosen architectural constants; no data-fitted free parameters appear because the system is training-free. The invented modules are internal scaffolding whose only evidence is the paper’s own diagnostics.

free parameters (3)
  • router top-K
    Number of vision experts selected per query; chosen by design and not fitted to the evaluation suites.
  • adapter depth L
    Number of stacked fusion blocks that integrate expert features; architectural hyper-parameter taken from the MoVA-style adapter.
  • Bloom step order and entry-level logic
    Fixed ordered list [LIST, EXPLAIN, ILLUSTRATE, COMPARE, HYPOTHESIZE, FINAL] and the rule that the query selects a starting index; hand-designed protocol, not learned.
axioms (3)
  • domain assumption Bloom’s educational taxonomy supplies a useful hierarchical decomposition of the cognitive demands of vision-language queries.
    Invoked throughout §3.2 and the design of the verbalizer; never validated against human cognitive data.
  • domain assumption Specialized CV experts produce only literal, non-inferential observations that can be decomposed into atomic, citeable statements without introducing reasoning.
    Stated as the goal of the Literal Evidence Summary stage (§3.2); success of later grounding depends on it.
  • domain assumption An LLM given natural-language expert descriptions plus coarse visual tokens can reliably select the relevant top-K experts.
    Router design in §3.4; no ablation of routing accuracy is reported.
invented entities (3)
  • Literal Evidence Summary / typed evidence store no independent evidence
    purpose: Normalize heterogeneous expert outputs into short, ID-indexed, non-inferential statements for citation.
    Core intermediate representation introduced in §3.2; independent evidence is only the paper’s own qualitative examples.
  • Bloom Verbalizer (f_Bloom) no independent evidence
    purpose: Generate a staged, evidence-cited draft answer following the fixed Bloom hierarchy.
    Defined by equation (4) and the step set L; its correctness is internal to the method.
  • Reasoning Trace Module no independent evidence
    purpose: Deterministically parse Bloom output and evidence store to recover entry level, executed steps and citation sets for quantitative analysis.
    Introduced in §3.3 solely as a diagnostic; no external validation of the recovered traces.

pith-pipeline@v1.1.0-grok45 · 19392 in / 2697 out tokens · 62959 ms · 2026-07-14T09:11:37.745545+00:00 · methodology

0 comments
read the original abstract

Large Vision Language Models (LVLMs) require strong reasoning over both visual and textual input. Recent work suggests that cognitive elements, especially diverse representations and metacognition, correlate with better performance. Many of the needed perceptual functions are already provided by specialized domain-specific computer vision models, which act as the perceptual subsystem for detecting objects, localizing them, inferring states, recovering spatial layout, and reading text. The key challenge is to integrate these multi-encoder experts into a trustworthy, interpretable, and coherent representation that improves verifiability and reduces hallucinations. This is difficult because vision-language questions span different cognitive levels, yet most LVLM pipelines apply the same perception-reasoning routing regardless of the demand of each query. We propose an evidence-driven multimodal reasoning framework that utilizes a Bloom-inspired taxonomy as a hierarchical reasoning protocol. The two-stage cognitive verbalization first produces a Literal Evidence Summary by decomposing expert outputs into short, atomic evidence statements. It then performs Bloom Verbalization to turn these evidence items into a staged reasoning trace, and a lightweight Reasoning Trace Module quantitatively analyzes the trace to make evidence usage and reasoning progression explicit. Through this integration, we observed several improvements in perception and reasoning abilities. Moreover, the trace module provides quantitative evidence that different queries induce different cognitive entry levels and evidence-use trajectories that enable fine-grained analysis.

Figures

Figures reproduced from arXiv: 2607.10796 by Ngai-Man Cheung, Robert Wijaya.

Figure 1
Figure 1. Figure 1: Our method use expert outputs that converted into a Literal Evidence [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Three qualitative examples of the proposed Literal Evidence Summary [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: One simple representation to illustrate the Bloom verbalization (LIST to [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Course Perception (CP), Logical Reasoning (LR), Relation Reasoning [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparing the scores and accuracies of dimensions related to real-world [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: An example to illustrate the Bloom verbalization (LIST to Final Answer) [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: An example to illustrate the Bloom verbalization (LIST to Final Answer) [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Another example to illustrate the Bloom verbalization (LIST to Final [PITH_FULL_IMAGE:figures/full_fig_p024_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: JSON schema of the evidences [PITH_FULL_IMAGE:figures/full_fig_p025_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 23 linked inside Pith

  1. [1]

    Nature Human Behaviour pp

    Akata, E., Schulz, L., Coda-Forno, J., Oh, S.J., Bethge, M., Schulz, E.: Playing repeated games with large language models. Nature Human Behaviour pp. 1–11 (2025)

    2025

  2. [2]

    Advances in neural information processing systems35, 23716– 23736 (2022)

    Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Men- sch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems35, 23716– 23736 (2022)

    2022

  3. [3]

    Artificial intelligence42(1), 7–49 (1990)

    Anderson, J.R., Boyle, C.F., Corbett, A.T., Lewis, M.W.: Cognitive modeling and intelligent tutoring. Artificial intelligence42(1), 7–49 (1990)

    1990

  4. [4]

    arXiv preprint arXiv:2309.16609 (2023)

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al.: Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

    Pith/arXiv arXiv 2023

  5. [5]

    Advances in neural information processing systems35, 32897– 32912 (2022)

    Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O.K., Aggarwal, K., Som, S., Piao, S., Wei, F.: Vlmo: Unified vision-language pre-training with mixture-of- modality-experts. Advances in neural information processing systems35, 32897– 32912 (2022)

    2022

  6. [6]

    arXiv preprint arXiv:2401.02954 (2024)

    Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al.: Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954 (2024)

    Pith/arXiv arXiv 2024

  7. [7]

    arXiv preprint arXiv:2410.20268 (2024)

    Binz, M., Akata, E., Bethge, M., Br¨ andle, F., Callaway, F., Coda-Forno, J., Dayan, P., Demircan, C., Eckstein, M.K., ´Eltet˝ o, N., et al.: Centaur: a foundation model of human cognition. arXiv preprint arXiv:2410.20268 (2024)

    Pith/arXiv arXiv 2024

  8. [8]

    arXiv preprint arXiv:2306.03917 (2023)

    Binz, M., Schulz, E.: Turning large language models into cognitive models. arXiv preprint arXiv:2306.03917 (2023)

    Pith/arXiv arXiv 2023

  9. [9]

    Proceedings of the National Academy of Sciences120(6), e2218523120 (2023)

    Binz, M., Schulz, E.: Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences120(6), e2218523120 (2023)

    2023

  10. [10]

    Handbook 1: Cognitive domain

    Bloom, B.S., et al.: Taxonomy of educational objectives: The classification of edu- cational goals. Handbook 1: Cognitive domain

  11. [11]

    Advances in neural information processing systems33, 1877–1901 (2020)

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems33, 1877–1901 (2020)

    1901

  12. [12]

    arXiv preprint arXiv:2303.127121(2023)

    Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y.T., Li, Y., Lundberg, S., et al.: Sparks of artificial general intelligence: early experiments with gpt-4 (2023). arXiv preprint arXiv:2303.127121(2023)

    Pith/arXiv arXiv 2023

  13. [13]

    In: European conference on computer vision

    Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End- to-end object detection with transformers. In: European conference on computer vision. pp. 213–229. Springer (2020)

    2020

  14. [14]

    Chiang, W.L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., Stoica, I., Xing, E.P.: Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality (March 2023),https://lmsys.org/ blog/2023-03-30-vicuna/

    2023

  15. [15]

    Nature human behaviour8(10), 1851–1863 (2024)

    Collins, K.M., Sucholutsky, I., Bhatt, U., Chandra, K., Wong, L., Lee, M., Zhang, C.E., Zhi-Xuan, T., Ho, M., Mansinghka, V., et al.: Building machines that learn and think with people. Nature human behaviour8(10), 1851–1863 (2024)

    2024

  16. [16]

    Advances in neural information processing systems36, 49250–49267 (2023)

    Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P.N., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems36, 49250–49267 (2023)

    2023

  17. [17]

    Wijaya and N.-M

    Doveh, S., Arbelle, A., Harary, S., Herzig, R., Kim, D., Cascante-Bonilla, P., Al- fassy, A., Panda, R., Giryes, R., Feris, R., et al.: Dense and aligned captions (dac) 16 R. Wijaya and N.-M. Cheung promote compositional reasoning in vl models. Advances in Neural Information Processing Systems36, 76137–76150 (2023)

    2023

  18. [18]

    arXiv preprint arXiv:2303.1013010(2023)

    Eloundou, T., Manning, S., Mishkin, P., Rock, D.: Gpts are gpts: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.1013010(2023)

    Pith/arXiv arXiv 2023

  19. [19]

    Journal of Machine Learning Research 23(120), 1–39 (2022)

    Fedus, W., Zoph, B., Shazeer, N.: Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23(120), 1–39 (2022)

    2022

  20. [20]

    arXiv preprint arXiv:2303.13988 (2023)

    Hagendorff, T., Dasgupta, I., Binz, M., Chan, S.C., Lampinen, A., Wang, J.X., Akata, Z., Schulz, E.: Machine psychology. arXiv preprint arXiv:2303.13988 (2023)

    Pith/arXiv arXiv 2023

  21. [21]

    Neural computation3(1), 79–87 (1991)

    Jacobs, R.A., Jordan, M.I., Nowlan, S.J., Hinton, G.E.: Adaptive mixtures of local experts. Neural computation3(1), 79–87 (1991)

    1991

  22. [22]

    arXiv preprint arXiv:2511.16660 (2025)

    Kargupta, P., Li, S.S., Wang, H., Lee, J., Chen, S., Ahia, O., Light, D., Griffiths, T.L., Kleiman-Weiner, M., Han, J., et al.: Cognitive foundations for reasoning and their manifestation in llms. arXiv preprint arXiv:2511.16660 (2025)

    arXiv 2025

  23. [23]

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Doll´ ar, P., Girshick, R.: Segment anything (2023),https://arxiv.org/abs/2304.02643

    Pith/arXiv arXiv 2023

  24. [24]

    In: European Conference on Computer Vision

    Lee, B.K., Park, B., Won Kim, C., Man Ro, Y.: Moai: Mixture of all intelligence for large language and vision models. In: European Conference on Computer Vision. pp. 273–302. Springer (2024)

    2024

  25. [25]

    Lee, K., Joshi, M., Turc, I., Hu, H., Liu, F., Eisenschlos, J., Khandelwal, U., Shaw, P., Chang, M.W., Toutanova, K.: Pix2struct: Screenshot parsing as pretraining for visual language understanding (2023),https://arxiv.org/abs/2210.03347

    Pith/arXiv arXiv 2023

  26. [26]

    arXiv preprint arXiv:2307.16125 (2023)

    Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., Shan, Y.: Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 (2023)

    Pith/arXiv arXiv 2023

  27. [27]

    Li, F., Zhang, R., Zhang, H., Zhang, Y., Li, B., Li, W., Ma, Z., Li, C.: Llava-next- interleave: Tackling multi-image, video, and 3d in large multimodal models (2024), https://arxiv.org/abs/2407.07895

    Pith/arXiv arXiv 2024

  28. [28]

    The Lancet Digital Health5(6), e333–e335 (2023)

    Li, H., Moon, J.T., Purkayastha, S., Celi, L.A., Trivedi, H., Gichoya, J.W.: Ethics of large language models in medicine and medical research. The Lancet Digital Health5(6), e333–e335 (2023)

    2023

  29. [29]

    In: Proceedings of the 2023 conference on empirical methods in natural language processing

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hal- lucination in large vision-language models. In: Proceedings of the 2023 conference on empirical methods in natural language processing. pp. 292–305 (2023)

    2023

  30. [30]

    arXiv preprint arXiv:2505.04921 (2025)

    Li, Y., Liu, Z., Li, Z., Zhang, X., Xu, Z., Chen, X., Shi, H., Jiang, S., Wang, X., Wang, J., et al.: Perception, reason, think, and plan: A survey on large multimodal reasoning models. arXiv preprint arXiv:2505.04921 (2025)

    Pith/arXiv arXiv 2025

  31. [31]

    IEEE Transactions on Multimedia (2026)

    Lin, B., Tang, Z., Ye, Y., Huang, J., Zhang, J., Pang, Y., Jin, P., Ning, M., Luo, J., Yuan, L.: Moe-llava: Mixture of experts for large vision-language models. IEEE Transactions on Multimedia (2026)

    2026

  32. [32]

    Liu, F., Eisenschlos, J.M., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Chen, W., Collier, N., Altun, Y.: Deplot: One-shot visual language reasoning by plot-to-table translation (2023),https://arxiv.org/abs/2212.10505

    Pith/arXiv arXiv 2023

  33. [33]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tun- ing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 26296–26306 (2024)

    2024

  34. [34]

    Advances in neural information processing systems36, 34892–34916 (2023) Mixture of Cognitive Experts in Large Vision-Language Models 17

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36, 34892–34916 (2023) Mixture of Cognitive Experts in Large Vision-Language Models 17

    2023

  35. [35]

    Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al.: Mmbench: Is your multi-modal model an all-around player? In: European conference on computer vision. pp. 216–233. Springer (2024)

    2024

  36. [36]

    arXiv preprint arXiv:2408.06292 (2024)

    Lu, C., Lu, C., Lange, R.T., Foerster, J., Clune, J., Ha, D.: The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292 (2024)

    Pith/arXiv arXiv 2024

  37. [37]

    MIT press (2010)

    Marr, D.: Vision: A computational investigation into the human representation and processing of visual information. MIT press (2010)

    2010

  38. [38]

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.Y., Li, S.W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: Dinov2: Learning robust visual featur...

    Pith/arXiv arXiv 2024

  39. [39]

    arXiv preprint arXiv:2308.00951 (2023)

    Puigcerver, J., Riquelme, C., Mustafa, B., Houlsby, N.: From sparse to soft mixtures of experts. arXiv preprint arXiv:2308.00951 (2023)

    Pith/arXiv arXiv 2023

  40. [40]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  41. [41]

    Advances in Neural Information Processing Systems34, 8583–8595 (2021)

    Riquelme, C., Puigcerver, J., Mustafa, B., Neumann, M., Jenatton, R., Su- sano Pinto, A., Keysers, D., Houlsby, N.: Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems34, 8583–8595 (2021)

    2021

  42. [42]

    Nature Machine Intelligence7(1), 96–106 (2025)

    Schulze Buschoff, L.M., Akata, E., Bethge, M., Schulz, E.: Visual cognition in mul- timodal large language models. Nature Machine Intelligence7(1), 96–106 (2025)

    2025

  43. [43]

    arXiv preprint arXiv:2303.07226 (2023)

    Shen, S., Yao, Z., Li, C., Darrell, T., Keutzer, K., He, Y.: Scaling vision-language models with sparse mixture of experts. arXiv preprint arXiv:2303.07226 (2023)

    Pith/arXiv arXiv 2023

  44. [44]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., Rohrbach, M.: Towards vqa models that can read. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8317–8326 (2019)

    2019

  45. [45]

    arXiv preprint arXiv:2312.1180510(2024)

    Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J., Yu, J., Soricut, R., Schalk- wyk, J., Dai, A., Hauth, A., et al.: Gemini: A family of highly capable multimodal models, 2024. arXiv preprint arXiv:2312.1180510(2024)

    Pith/arXiv arXiv 2024

  46. [46]

    arXiv preprint arXiv:2302.13971 (2023)

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozi` ere, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    Pith/arXiv arXiv 2023

  47. [47]

    In: Parsing the Turing test: Philosophical and methodological issues in the quest for the thinking computer, pp

    Turing, A.M.: Computing machinery and intelligence. In: Parsing the Turing test: Philosophical and methodological issues in the quest for the thinking computer, pp. 23–65. Springer (2007)

    2007

  48. [48]

    Wei, H., Kong, L., Chen, J., Zhao, L., Ge, Z., Yang, J., Sun, J., Han, C., Zhang, X.: Vary: Scaling up the vision vocabulary for large vision-language models (2023), https://arxiv.org/abs/2312.06109

    Pith/arXiv arXiv 2023

  49. [49]

    Communications of the ACM9(1), 36–45 (1966)

    Weizenbaum, J.: Eliza—a computer program for the study of natural language communication between man and machine. Communications of the ACM9(1), 36–45 (1966)

    1966

  50. [50]

    arXiv e-prints pp

    Wijaya, R., Nguyen, N.B., Cheung, N.M.: Multimodal preference data synthetic alignment with reward model. arXiv e-prints pp. arXiv–2412 (2024)

    2024

  51. [51]

    arXiv preprint arXiv:2308.02490 (2023) 18 R

    Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., Wang, L.: Mm- vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490 (2023) 18 R. Wijaya and N.-M. Cheung

    Pith/arXiv arXiv 2023

  52. [52]

    IEEE transactions on neural networks and learning systems23(8), 1177–1193 (2012)

    Yuksel, S.E., Wilson, J.N., Gader, P.D.: Twenty years of mixture of experts. IEEE transactions on neural networks and learning systems23(8), 1177–1193 (2012)

    2012

  53. [53]

    org/abs/2303.00915

    Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., Wong, C., Tupini, A., Wang, Y., Mazzola, M., Shukla, S., Liden, L., Gao, J., Crabtree, A., Piening, B., Bifulco, C., Lungren, M.P., Naumann, T., Wang, S., Poon, H.: Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million sc...

    Pith/arXiv arXiv 2025

  54. [54]

    arXiv preprint arXiv:2509.25373 (2025)

    Zhou, C., Wang, M., Ma, Y., Wu, C., Chen, W., Qian, Z., Liu, X., Zhang, Y., Wang, J., Xu, H., et al.: From perception to cognition: A survey of vision- language interactive reasoning in multimodal large language models. arXiv preprint arXiv:2509.25373 (2025)

    arXiv 2025

  55. [55]

    Advances in Neural Information Processing Systems37, 103305–103333 (2024)

    Zong, Z., Ma, B., Shen, D., Song, G., Shao, H., Jiang, D., Li, H., Liu, Y.: Mova: Adapting mixture of vision experts to multimodal context. Advances in Neural Information Processing Systems37, 103305–103333 (2024)

    2024

  56. [56]

    Table 4a shows the distribution of text-token counts in TextVQA, while Table 4b presents the corresponding distribution for MME

    Zong, Z., Song, G., Liu, Y.: Detrs with collaborative hybrid assignments training (2023),https://arxiv.org/abs/2211.12860 Mixture of Cognitive Experts in Large Vision-Language Models 19 A Appendix / Supplementary Materials A.1 T ext Evidence The resulting evidence annotations also allow us to examine text patterns within each dataset. Table 4a shows the d...

    Pith/arXiv arXiv 2023