Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Structured demonstration assembly under a six-level cognitive taxonomy is the main driver of stable unified multimodal in-context learning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 18:41 UTC pith:PNJJMSYR

load-bearing objection Real infrastructure paper: taxonomy + 760K curated 8-shot episodes are the load-bearing contribution; CAPM is secondary, and external transfer is still the open check. the 4 major comments →

arxiv 2603.24690 v3 pith:PNJJMSYR submitted 2026-03-25 cs.CV

UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy

Yicheng Xu , Jiangning Zhang , Zhucun Xue , Teng Hu , Ran Yi , Xiaobin Hu , Yong Liu , Dacheng Tao This is my paper
classification cs.CV
keywords multimodal in-context learningunified multimodal modelscapability-oriented taxonomyfew-shot learningdemonstration assemblyUniICL-760KContext-Adaptive Prototype Modulatornon-monotonic shot scaling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

In-context learning lets multimodal models adapt from a few examples without weight updates, but it is fragile: adding shots can help or hurt depending on the task, and interleaving images with text often injects interference. This paper claims the root cause is treating every demonstration as interchangeable, ignoring what cognitive job it is meant to do. The authors sort tasks into six levels—from basic perception up to high-order discernment—and use that map to curate UniICL-760K, a large set of carefully matched 8-shot episodes, plus a controlled benchmark. They show that assembling demos by cognitive role, not model size or raw shot count, is what mainly produces consistent gains; a small plug-in modulator further stabilizes few-shot behavior. The result is competitive unified performance on both understanding and generation, beating larger multimodal baselines on most understanding ICL tasks.

Core claim

The primary source of gains in unified multimodal in-context learning is data-driven assembly of demonstrations guided by a six-level capability-oriented taxonomy that ranks the functional role of examples from perception through discernment. When episodes are curated this way into UniICL-760K and evaluated on UniICL-Bench, the approach yields highly competitive unified results and outperforms larger-parameter multimodal baselines on most understanding ICL tasks, with a lightweight Context-Adaptive Prototype Modulator providing complementary few-shot stability.

What carries the argument

Six-level Capability-Oriented Taxonomy (perception, imitation, conception, deduction, analogy, discernment) that defines the functional role of demonstrations and drives curation of UniICL-760K; complemented by the Context-Adaptive Prototype Modulator (CAPM), a plug-and-play module that disentangles demonstration encoding and dynamically routes context to stabilize adaptation.

Load-bearing premise

The automated cascade of tagging, segmentation, captioning, filtering, and generative synthesis produces demonstration–query pairs that truly reflect the intended cognitive levels rather than artifacts of the synthesis models themselves.

What would settle it

Replace taxonomy-matched demonstrations on UniICL-Bench with equal-sized random or purely visually similar sets, or re-evaluate the same models on a fully human-curated subset of the same tasks; if the reported gains and more monotonic shot scaling disappear, the claim that capability-oriented assembly is the primary source of the gains is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Unified models can be made few-shot stable by matching demo cognitive load rather than by simply scaling model size or shot count.
  • Non-monotonic shot scaling can be diagnosed and mitigated once tasks are labeled by the functional role of demonstrations.
  • Training corpora organized by cognitive role of demos transfer better across understanding and generation than modality- or task-specific ICL sets.
  • A lightweight routing module on top of standard attention is enough to reduce cross-modal interference in dense 8-shot contexts.
  • Controlled multi-shot benchmarks become necessary to expose when more demos help versus hurt.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same taxonomy could be used to re-score existing multimodal benchmarks and reveal which ones actually test high-order analogy versus low-level perception.
  • If generation remains more sensitive to demo quality than understanding, future unified architectures may need separate context pathways for generative versus discriminative heads.
  • Extending the curation pipeline to video or audio would test whether the six levels remain stable when temporal structure is added.
  • Human re-annotation of a UniICL-Bench subset would quantify how much measured gains depend on synthetic alignment artifacts.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses non-monotonic, task-dependent few-shot behavior in unified multimodal models that jointly handle understanding and generation. It proposes a six-level Capability-Oriented Taxonomy (perception, imitation, conception, deduction, analogy, discernment) that classifies demonstrations by cognitive role, then uses this taxonomy to curate UniICL-760K (~766k 8-shot episodes over 15 subtasks) and UniICL-Bench. As a secondary architectural stabilizer it introduces CAPM, a lightweight plug-and-play context-adaptive prototype modulator. The central empirical claim is that taxonomy-guided demonstration assembly is the primary source of gains; CAPM further improves few-shot stability. On UniICL-Bench the approach is reported to be competitive with unified baselines and to outperform larger-parameter MLLMs on most understanding ICL tasks, with supporting k-shot curves, perturbation/stability tests, and qualitative generation comparisons.

Significance. If the results hold under external scrutiny, the work would supply the field with a cognitively structured training corpus and evaluation suite for unified multimodal ICL, plus a diagnostic lens for non-monotonic shot scaling that existing modality- or task-specific benchmarks largely miss. Explicitly attributing gains primarily to data assembly rather than the auxiliary module is a useful scientific stance. The perturbation suite (random replacement, reverse ordering, noisy demos) and forward-feature control study are concrete, falsifiable probes of context sensitivity. Code and data release plans further raise the work’s potential utility. The main significance risk is that headline outperformance may partly reflect match to a synthetic curation distribution rather than transferable ICL capability.

major comments (4)
  1. The headline claim that the method “outperforms larger-parameter MLLM baselines on most understanding ICL tasks” (abstract; §1; UniICL-Bench evaluations) is evaluated almost entirely on UniICL-Bench, which is derived from the same UniICL-760K collection and curation pipeline (§3.2.2). Cascaded annotation and synthesis rely on Qwen3-VL variants, GLM-4.5V filtering, and related generative editors—models in the same family as several reported baselines—while many metrics are MLLM-as-judge scores. This creates a load-bearing self-consistency risk: measured gains may partly reflect distributional match to the synthetic style rather than genuine capability-level ICL. At least one external ICL transfer suite (e.g., VL-ICL Bench or held-out non-synthetic understanding/generation ICL sets) with the same shot protocol is needed to separate pipeline artifacts from transferable gains.
  2. The paper asserts that “data-driven assembly is the primary source of our gains” and treats CAPM as a complementary stabilizer (abstract; contributions). The manuscript does not, in the provided text, report a clean factorial ablation (taxonomy-guided UniICL-760K training with vs. without CAPM; CAPM on a non-taxonomy or randomly assembled baseline; zero-shot vs. k-shot under matched data) with effect sizes on the same UniICL-Bench splits. Without that table, the primary-vs-auxiliary attribution remains under-supported relative to the strength of the claim. A compact ablation table quantifying Δ from data assembly alone vs. CAPM alone vs. both is required.
  3. §3.2.2 and the Limitations note that automated synthesis and external models may introduce biases, yet UniICL-Bench still uses the same episode construction (dense captions, scene graphs, DPP/intent retrieval, generative edits). For high-order levels (analogy, discernment) and generative subtasks (visual refinement, analogical editing, chain-of-editing), there is no reported human agreement study or independent label audit on demonstration–query alignment and CoT quality. Without a quantified fidelity check on a stratified sample of episodes, it is hard to know whether k-shot curves (Fig. 6) and qualitative wins (Figs. 8, 10) measure cognitive ICL or residual pipeline regularities. A small human validation or inter-annotator agreement report on alignment correctness by taxonomy level would substantially strengthen the central claim.
  4. The six-level taxonomy is motivated by neurocognitive citations and used to structure both data and evaluation (Fig. 1; §1). The manuscript does not show that the level assignments are reproducible (e.g., inter-rater or automatic classifier agreement) or that non-monotonic scaling patterns cleanly stratify by level as predicted (perception hindered by extra shots; deduction/analogy helped). Fig. 6 shows selected tasks but not a full level-wise shot-scaling summary. Either a level-stratified scaling table or an explicit statement that level labels are design choices rather than validated cognitive measurements is needed so readers can interpret the diagnostic claims.
minor comments (5)
  1. Several figure panels and mid-paper passages appear corrupted or OCR-garbled in the submitted text (e.g., Fig. 1 caption fragments, Fig. 5 CAPM pipeline description, equation-like tokens in Fig. 7). Clean camera-ready figures and a complete CAPM four-stage formalization (inputs, prototype construction, routing, residual injection) are needed for reproducibility.
  2. Abstract and intro state data/code “available” / “will be available soon” at the GitHub and Hugging Face links; ensure the camera-ready version freezes the exact UniICL-760K release and CAPM implementation used for the reported numbers.
  3. Clarify evaluation metrics per subtask (mIoU, MLLM-Score, Eff., HPSv3, CLIPScore, GenEval, etc.) in one table so understanding vs. generation comparisons are not mixed under a single “MLLM-Score” label.
  4. Fig. 9 failure case (spatial grounding under overlapping multi-shot edits) is useful; briefly discuss whether CAPM or longer-context retrieval is intended to address it, or leave it as an open limitation.
  5. Related-work coverage of multimodal ICL (Flamingo, Otter, MMICL, VisualCloze, VL-ICL Bench) is present but could more sharply contrast taxonomy-guided episode construction against prior similarity-only retrieval.

Circularity Check

0 steps flagged

No significant circularity: empirical dataset/module claims do not reduce by construction to their inputs.

full rationale

This is an empirical systems paper (taxonomy + curated ICL corpus + optional CAPM module + UniICL-Bench evals), not a first-principles derivation. The load-bearing claim—that taxonomy-guided data-driven assembly of 8-shot episodes is the primary source of gains, with CAPM a lightweight stabilizer—is supported by ablations, k-shot curves, and perturbation studies (random replacement, reverse ordering, noisy demos; Figs. 6–7, 11), not by equating a fitted quantity to a “prediction.” The six-level taxonomy is an organizational framework inspired by external neurocognitive citations, not a uniqueness theorem imported from the authors. Training/evaluating on UniICL-760K/UniICL-Bench and using large VLMs in the curation pipeline raise distributional-validity concerns (flagged in Limitations), but that is a confound risk, not circularity: no equation or result is definitionally forced by the inputs, and comparisons are to external model families on the proposed bench. No self-definitional loop, fitted-input-as-prediction, load-bearing self-citation uniqueness claim, or ansatz smuggled via self-citation is present. Score 0 is therefore appropriate.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 4 invented entities

This is an empirical systems paper. Load-bearing premises are domain assumptions about ICL sensitivity and cross-modal interference, the neurocognitive-inspired taxonomy as a useful organizing axis, and the validity of automated multi-model curation plus MLLM-as-judge style scoring. Free parameters are design choices (shot count, taxonomy depth, CAPM internals, retrieval/DPP settings). Invented entities are the taxonomy, dataset/bench, and CAPM—not physical objects, but new constructs the claims depend on.

free parameters (5)
  • taxonomy_depth_and_level_definitions
    Six levels (perception→discernment) and their task mappings are author-chosen partitions inspired by cognitive literature, not uniquely determined by data.
  • max_shots_in_episodes
    8-shot episode design is a fixed design choice used for corpus and bench construction.
  • CAPM_architecture_and_routing_hyperparameters
    Prototype modulator stages, routing, and any scale factors are model design knobs fitted/selected for few-shot stability.
  • DPP_and_retrieval_sampling_settings
    Multi-modal feature fusion and Determinantal Point Process sampling parameters control demonstration diversity/compatibility and thus measured ICL gains.
  • quality_filter_thresholds_in_curation
    Cascaded filtering from ~750k raw images down to ~202k retained scenes depends on model-based hallucination filters and quality cutoffs.
axioms (5)
  • domain assumption In-context learning efficacy in unified multimodal models is highly sensitive to demonstration selection/formatting and can be non-monotonic in shot count due to cross-modal interference.
    Stated as motivating premise in abstract and §1; supported by citations and their curves, but treated as background fact for the program.
  • ad hoc to paper Demonstration functional roles can be usefully ordered into six cognitive levels inspired by neurocognitive development (perception, imitation, conception, deduction, analogy, discernment).
    Core organizing axiom of the taxonomy (§1, Fig. 1); maps loosely to cited cognitive literature but is a paper-specific operationalization.
  • domain assumption Automated dense annotation and generative synthesis with large VLMs/expert editors yield sufficiently high-quality ICL episodes for training and evaluation.
    Pipeline §3.2.2 depends on this; limitations section admits bias/reproducibility risk.
  • domain assumption MLLM-based automatic scores and controlled UniICL-Bench metrics are adequate proxies for understanding/generation ICL quality.
    Used throughout evaluation figures (e.g., MLLM-Score curves); standard but imperfect in the field.
  • domain assumption Standard transformer self-attention over interleaved multimodal tokens is a valid base that CAPM can stabilize without changing the overall training-free ICL interface.
    Architectural premise for CAPM as plug-and-play module (Fig. 5).
invented entities (4)
  • Six-level Capability-Oriented Taxonomy for multimodal ICL no independent evidence
    purpose: Categorize functional roles of demonstrations to diagnose non-monotonic shot scaling and guide dataset construction.
    Author-defined cognitive ladder; independent evidence is only empirical usefulness on their tasks, not an external validated cognitive instrument.
  • UniICL-760K independent evidence
    purpose: Large-scale curated 8-shot multimodal ICL training corpus across 15 subtasks spanning understanding and generation.
    New dataset artifact; value is instrumental and measured on the companion bench.
  • UniICL-Bench no independent evidence
    purpose: Controlled evaluation suite for multi-dimensional ICL capability and stability up to 8 shots.
    New benchmark; external validity beyond this paper is not yet established.
  • Context-Adaptive Prototype Modulator (CAPM) no independent evidence
    purpose: Lightweight plug-and-play module to disentangle demonstration encoding and stabilize few-shot adaptation.
    New architectural component; evidence is ablation/performance on UniICL-Bench only.

pith-pipeline@v1.1.0-grok45 · 14611 in / 3824 out tokens · 46036 ms · 2026-07-13T18:41:12.447640+00:00 · methodology

0 comments
read the original abstract

In-context learning (ICL) enables fast task adaptation from demonstrations without per-task parameter updates but remains highly sensitive to example selection and formatting. In unified multimodal models spanning understanding and generation, this sensitivity is exacerbated by cross-modal interference and varying cognitive demands. Consequently, in-context learning efficacy is often non-monotonic and highly task-dependent. To diagnose these behaviors, we introduce a six-level Capability-Oriented Taxonomy that categorizes the functional role of demonstrations from basic perception to high-order discernment. Guided by this cognitive framework, we construct UniICL-760K, a large-scale corpus featuring curated 8-shot in-context learning episodes across 15 subtasks, alongside UniICL-Bench for rigorous, controlled evaluation. We show that this data-driven assembly is the primary source of our gains. As a complementary, lightweight stabilizer, we additionally propose the Context-Adaptive Prototype Modulator, a plug-and-play module that further improves few-shot stability. Evaluations on UniICL-Bench show that our approach yields highly competitive unified results, outperforming larger-parameter multimodal large language model baselines on most understanding in-context learning tasks. Data and code are available at https://github.com/xuyicheng-zju/UniICL.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ChatImage: Navigating Long-Form LLM Answers through Interactive Images

    cs.CV 2026-07 conditional novelty 5.0

    ChatImage renders LLM answers as images, then uses visual grounding to place clickable hotspots on rendered regions for interactive follow-up.

Reference graph

Works this paper leans on

53 extracted references · 22 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 2022. 1, 3, 5

    2022

  2. [2]

    Openflamingo: An open-source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390, 2023

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bit- ton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390, 2023. 5

    Pith/arXiv arXiv 2023

  3. [3]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025. 5, 9

    Pith/arXiv arXiv 2025

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind 17 Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020. 1, 3

    2020

  5. [5]

    Can multimodal large language models truly perform multimodal in-context learning? In2025 IEEE/CVF Winter Conference on Applications of Computer Vision (W ACV), pages 6000–6010

    Shuo Chen, Zhen Han, Bailan He, Jianzhe Liu, Mark Buckley, Yao Qin, Philip Torr, Volker Tresp, and Jindong Gu. Can multimodal large language models truly perform multimodal in-context learning? In2025 IEEE/CVF Winter Conference on Applications of Computer Vision (W ACV), pages 6000–6010. IEEE, 2025. 1

    2025

  6. [6]

    Levels of processing: A framework for memory research.Journal of verbal learning and verbal behavior, 1972

    Fergus IM Craik and Robert S Lockhart. Levels of processing: A framework for memory research.Journal of verbal learning and verbal behavior, 1972. 2, 4

    1972

  7. [7]

    Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683,

    Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683,

    Pith/arXiv arXiv

  8. [8]

    Dreamllm: Synergistic multimodal comprehension and creation.arXiv preprint arXiv:2309.11499, 2023

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation.arXiv preprint arXiv:2309.11499, 2023. 1, 3

    Pith/arXiv arXiv 2023

  9. [9]

    Distributed hierarchical processing in the primate cerebral cortex

    Daniel J Felleman and David C Van Essen. Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex (New York, NY: 1991), 1991. 2, 4

    1991

  10. [10]

    Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 2023

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 2023. 6, 8

    2023

  11. [11]

    Processing capacity defined by relational complexity: Implications for comparative, developmental, and cognitive psychology.Behavioral and brain sciences, 1998

    Graeme S Halford, William H Wilson, and Steven Phillips. Processing capacity defined by relational complexity: Implications for comparative, developmental, and cognitive psychology.Behavioral and brain sciences, 1998. 2, 4

    1998

  12. [12]

    Clipscore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. InEMNLP, 2021. 7

    2021

  13. [13]

    Glm-4.5 v and glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning.arXiv preprint arXiv:2507.01006, 2025

    Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.5 v and glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning.arXiv preprint arXiv:2507.01006, 2025. 5

    Pith/arXiv arXiv 2025

  14. [14]

    Detect anything via next point prediction, 2025

    Qing Jiang, Junan Huo, Xingyu Chen, Yuda Xiong, Zhaoyang Zeng, Yihao Chen, Tianhe Ren, Junzhi Yu, and Lei Zhang. Detect anything via next point prediction, 2025. URLhttps://arxiv.org/abs/2510.12798. 5

    arXiv 2025

  15. [15]

    Generating images with multimodal language models

    Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. NeurIPS, 2023. 1, 3

    2023

  16. [16]

    What matters when building vision-language models?NeurIPS, 2024

    Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?NeurIPS, 2024. 3

    2024

  17. [17]

    Mimic-it: Multi-modal in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. 2023. 1

    2023

  18. [18]

    Otter: A multi-modal model with in-context instruction tuning.T-PAMI, 2025

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Joshua Adrian Cahyono, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning.T-PAMI, 2025. 1

    2025

  19. [19]

    Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023. 1, 3, 8

    Pith/arXiv arXiv 2023

  20. [20]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InICML, 2023. 1, 3

    2023

  21. [21]

    M2iv: Towards efficient and fine-grained multimodal in-context learning in large vision-language models.arXiv e-prints, pages arXiv–2504,

    Yanshu Li, Hongyang He, Yi Cao, Qisen Cheng, Xiang Fu, and Ruixiang Tang. M2iv: Towards efficient and fine-grained multimodal in-context learning in large vision-language models.arXiv e-prints, pages arXiv–2504,

  22. [22]

    Visualcloze: A universal image generation framework via visual in-context learning

    Zhong-Yu Li, Ruoyi Du, Juncheng Yan, Le Zhuo, Zhen Li, Peng Gao, Zhanyu Ma, and Ming-Ming Cheng. Visualcloze: A universal image generation framework via visual in-context learning. InICCV, 2025. 1, 3

    2025

  23. [23]

    Uniworld-v1: High-resolution semantic encoders for unified visual understanding and generation.arXiv preprint arXiv:2506.03147, 2025

    Bin Lin, Zongjian Li, Xinhua Cheng, Yuwei Niu, Yang Ye, Xianyi He, Shenghai Yuan, Wangbo Yu, Shaodong Wang, Yunyang Ge, et al. Uniworld-v1: High-resolution semantic encoders for unified visual understanding and generation.arXiv preprint arXiv:2506.03147, 2025. 3, 9

    Pith/arXiv arXiv 2025

  24. [24]

    Mmbench: Is your multi-modal model an all-around player? InECCV, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InECCV, 2024. 1, 3, 6, 8 18

    2024

  25. [25]

    The flan collection: Designing data and methods for effective instruction tuning

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. InICML,

  26. [26]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. InACL, 2022. 1

    2022

  27. [27]

    Hpsv3: Towards wide-spectrum human preference score

    Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. InICCV, 2025. 6, 7

    2025

  28. [28]

    In-context learning and induction heads.arXiv preprint arXiv:2209.11895, 2022

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads.arXiv preprint arXiv:2209.11895, 2022. 3

    Pith/arXiv arXiv 2022

  29. [29]

    What factors affect multi-modal in-context learning? an in-depth exploration.Advances in Neural Information Processing Systems, 2024

    Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Li, and Wanxiang Che. What factors affect multi-modal in-context learning? an in-depth exploration.Advances in Neural Information Processing Systems, 2024. 1

    2024

  30. [30]

    Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free.arXiv preprint arXiv:2505.06708, 2025

    Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, et al. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free.arXiv preprint arXiv:2505.06708, 2025. 9

    Pith/arXiv arXiv 2025

  31. [31]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024. 5

    Pith/arXiv arXiv 2024

  32. [32]

    Laion-5b: An open large-scale dataset for training next generation image-text models.NeurIPS, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models.NeurIPS, 2022. 5

    2022

  33. [33]

    Dinov3.arXiv preprint arXiv:2508.10104, 2025

    Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3.arXiv preprint arXiv:2508.10104, 2025. 6

    Pith/arXiv arXiv 2025

  34. [34]

    Emu: Generative pretraining in multimodality.arXiv preprint arXiv:2307.05222,

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality.arXiv preprint arXiv:2307.05222,

    Pith/arXiv arXiv

  35. [35]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. InCVPR, 2024. 3

    2024

  36. [36]

    Codi-2: In-context interleaved and interactive any-to-any generation

    Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal. Codi-2: In-context interleaved and interactive any-to-any generation. InCVPR, 2024. 3

    2024

  37. [37]

    Determinantal point processes for machine learning.stat, 2013

    Alex Kulesza Ben Taskar. Determinantal point processes for machine learning.stat, 2013. 6

    2013

  38. [38]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818,

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818,

    Pith/arXiv arXiv

  39. [39]

    Qwen2.5-vl, January 2025

    Qwen Team. Qwen2.5-vl, January 2025. URLhttps://qwenlm.github.io/blog/qwen2.5-vl/. 9

    2025

  40. [40]

    Transformers learn in-context by gradient descent

    Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. InICML, 2023. 3

    2023

  41. [41]

    Ovis-u1 technical report.arXiv preprint arXiv:2506.23044, 2025

    Guo-Hua Wang, Shanshan Zhao, Xinjie Zhang, Liangfu Cao, Pengxin Zhan, Lunhao Duan, Shiyin Lu, Minghao Fu, Xiaohao Chen, Jianshan Zhao, et al. Ovis-u1 technical report.arXiv preprint arXiv:2506.23044, 2025. 3, 9

    Pith/arXiv arXiv 2025

  42. [42]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025. 9

    Pith/arXiv arXiv 2025

  43. [43]

    Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869,

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869,

    Pith/arXiv arXiv

  44. [44]

    Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. InEMNLP, 2022. 3 19

    2022

  45. [45]

    Q-align: Teaching lmms for visual scoring via discrete text-defined levels.arXiv preprint arXiv:2312.17090, 2023

    Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al. Q-align: Teaching lmms for visual scoring via discrete text-defined levels.arXiv preprint arXiv:2312.17090, 2023. 6, 7

    Pith/arXiv arXiv 2023

  46. [46]

    An explanation of in-context learning as implicit bayesian inference.arXiv preprint arXiv:2111.02080, 2021

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference.arXiv preprint arXiv:2111.02080, 2021. 3

    Pith/arXiv arXiv 2021

  47. [47]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InCVPR, 2024. 1, 3, 6, 8

    2024

  48. [48]

    Nexus-gen: Unified image understanding, generation, and editing via prefilled autoregression in shared embedding space.arXiv preprint arXiv:2504.21356, 2025

    Hong Zhang, Zhongjie Duan, Xingjun Wang, Yuze Zhao, Weiyi Lu, Zhipeng Di, Yixuan Xu, Yingda Chen, and Yu Zhang. Nexus-gen: Unified image understanding, generation, and editing via prefilled autoregression in shared embedding space.arXiv preprint arXiv:2504.21356, 2025. 3, 9

    Pith/arXiv arXiv 2025

  49. [49]

    Qwen3 embedding: Advancing text embedding and reranking through foundation models

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025. 6

    Pith/arXiv arXiv 2025

  50. [50]

    Recognize anything: A strong image tagging model

    Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. InCVPR, 2024. 5

    2024

  51. [51]

    Mmicl: Empowering vision-language model with multi-modal in-context learning.arXiv preprint arXiv:2309.07915, 2023

    Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning.arXiv preprint arXiv:2309.07915, 2023. 1

    Pith/arXiv arXiv 2023

  52. [52]

    Judging llm-as-a-judge with mt-bench and chatbot arena.NeurIPS, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.NeurIPS, 2023. 7

    2023

  53. [53]

    Vl-icl bench: The devil in the details of multimodal in-context learning.arXiv preprint arXiv:2403.13164, 2024

    Yongshuo Zong, Ondrej Bohdal, and Timothy Hospedales. Vl-icl bench: The devil in the details of multimodal in-context learning.arXiv preprint arXiv:2403.13164, 2024. 7, 8 20

    Pith/arXiv arXiv 2024