Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

A frozen large vision-language model can be personalized for multiple new visual concepts in a single forward pass, without inference-time training, by converting reference images into adaptive-length in-context prompts and label embeddings

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 12:43 UTC pith:4SXZQ6LV

load-bearing objection Credible and well-ablated personalization method, but the theoretical support for its key regularizer is tautological and the empirical SOTA claim is unauditable without code/data. the 4 major comments →

arxiv 2605.31513 v2 pith:4SXZQ6LV submitted 2026-05-29 cs.CV

Personalize Your Large Vision-language Models With In-context Prompt Tuning

Yanshu Li , Jiaqian Li , Kuai Yu , Xi Xiao , Dongfang Liu , Tianyang Wang , Ruixiang Tang This is my paper
classification cs.CV
keywords LVLM personalizationin-context prompt tuningsoft promptsmulti-concept personalizationmulti-image personalizationgeometric regularizationdynamic token routingfrozen vision-language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that LVLM personalization need not require per-concept fine-tuning or test-time training. It proposes in-context prompt tuning (ICPT), which uses a lightweight projector to turn a few reference images of each new concept into continuous prompts plus label embeddings that a frozen LVLM can reason about directly. Two geometric regularizers, Contextual Variation Memory and Margin-constrained Concept Separation, are designed to keep prompts focused on identity and remove cross-concept confusion. The paper reports state-of-the-art accuracy on four personalization tasks across four LVLM backbones, including challenging multi-image, multi-concept settings, while using fewer tokens per concept than fixed-length baselines.

Core claim

ICPT claims to achieve state-of-the-art personalization accuracy by simulating in-context learning inside the latent space of a frozen LVLM. For each concept, the Adaptive Concept Projector extracts hierarchical features from the LVLM's own vision encoder, aggregates them across reference images, and produces a dedicated label embedding plus a variable-length visual prompt. The Dynamic Token Router prunes this prompt down to the visual complexity of the concept, and the two geometric constraints — Contextual Variation Memory, which penalizes prompt components aligned with environment-induced variation directions, and Margin-constrained Concept Separation, which bounds cross-concept similarit

What carries the argument

The central object is the Adaptive Concept Projector (ACP): a single-layer cross-attention module plus MLP that distills multi-scale visual features from the LVLM's native vision encoder into a label embedding and a full-capacity visual prompt. The Dynamic Token Router assigns a per-token score and prunes low-scoring tokens at inference. Contextual Variation Memory stores normalized pairwise differences of same-concept reference prompts, treated as environment-induced variation directions, and penalizes the prompt's projection onto them. Margin-constrained Concept Separation imposes a soft upper bound m on cosine similarity between prompts and between label embeddings, preserving shared sema

Load-bearing premise

The load-bearing premise is that two reference photos of the same concept differ only in surroundings, so their difference represents pure environment; if pose, lighting, expression, or appearance changes do not cancel between photos, the environmental penalty can push away identity information along with background.

What would settle it

Build reference sets of the same identity with deliberate pose, lighting, and viewpoint variations and compare the direction of pairwise prompt differences to those from different identities: if the same-identity differences align significantly with cross-identity differences in the prompt space, the CVM penalty cannot separate environment from identity, and recognition accuracy on identity-sensitive queries should drop as variation increases.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • New concepts can be added to a deployed LVLM on the fly with a single forward pass and no fine-tuning, making personalization practical for interactive assistants and agents.
  • Prompt length adapts automatically to concept complexity, so simpler concepts consume fewer tokens and reduce latency in multi-concept contexts.
  • The method transfers across LVLM architectures and sizes, suggesting it can ride along as foundation models improve rather than requiring bespoke adaptation.
  • Multi-image, multi-concept reasoning tasks, such as recognizing and describing several user-specific people or objects in one query image, become feasible without retraining.
  • The geometric regularizers shift personalization from memorizing reference pixels to aligning identity-level semantics, which should generalize better to unseen environments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the identity/environment decomposition holds, a similar orthogonal-projection regularization could be applied to general multimodal in-context learning, stripping spurious correlations from demonstration images beyond personalization. This is an extension the paper does not make.
  • The paper's own appendix failure cases show that visually similar concepts still get confused and that prompts with many simultaneous labels strain the fixed template; the claimed robustness is therefore bounded by those settings, and hierarchical or compositional separation might be a testable improvement.
  • The paper's volume-versus-diversity experiment suggests that small, high-diversity training sets may be more valuable than large homogeneous ones; this could lower data collection costs for future personalization modules, a consequence the authors only sketch.
  • A testable next step would be to apply ICPT to concepts with continuous rather than discrete labels, or to dynamic concept updates over time, where the CVM memory could be maintained online instead of during a fixed training phase.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes In-Context Prompt Tuning (ICPT) for personalizing frozen LVLMs without inference-time training. An Adaptive Concept Projector (ACP) extracts hierarchical features from reference images using the LVLM's own vision encoder and produces a label embedding plus a variable-length visual prompt whose length is chosen by a Dynamic Token Router (DTR). Two geometric regularizers are introduced: Contextual Variation Memory (CVM), which penalizes prompt projection onto pairwise reference-image differences, and Margin-constrained Concept Separation (MCS), which bounds cross-concept similarity in visual and label spaces. The method is trained on a privately constructed dataset of 350 concepts and evaluated on 200 held-out concepts across four LVLMs, with results claimed to be state-of-the-art against ICL, GPT-4o+ICL, PLVM, MC-LLaVA, and other baselines on recognition, multiple-choice VQA, open-ended VQA, and captioning tasks in both single- and multi-concept settings.

Significance. The problem is timely and the proposed architecture has several appealing properties: it avoids per-concept fine-tuning at inference, handles multi-image and multi-concept inputs, adaptively allocates prompt length, and is evaluated across four LVLM backbones. The authors also ablate the main components and check sensitivity to hyperparameters. However, the claimed SOTA rests on a private, unreleased benchmark and single-run numbers, and the theoretical justification for CVM is a tautological bound that does not establish environmental disentanglement. If these issues are resolved, the method could be a useful step toward scalable LVLM personalization; as it stands, the empirical and theoretical support is insufficient for the strength of the claims.

major comments (4)
  1. [§3.4 and Appendix B, Theorem 1] Theorem 1 does not provide the claimed 'state-induced interference bound.' After enforcing the orthogonality constraint, Eq. (8) reduces to |<P_tilde, δ_perp>| ≤ ||δ_perp||, which is just Cauchy–Schwarz for any unit-norm matrix. It holds regardless of the CVM constraint and does not require span(M_ext) to be environment-only. The paper does not bound ||δ_perp|| or justify that δ_state has a small out-of-span component. Thus the theorem is not evidence that CVM 'neutralizes' environmental interference; the decomposition in Eq. (6) is definitional. The authors should either provide a nontrivial bound (e.g., in terms of training-time variation) or clearly state that CVM is an empirical regularizer without a theoretical guarantee.
  2. [§3.4, Eq. (4)] The load-bearing premise of CVM is that the difference P_full_{i,x} − P_full_{i,y} is an 'environment-induced variation,' with identity components canceling. However, the reference images are explicitly constructed to vary in pose, viewpoint, appearance, and background (Appendix A.1). In a ViT latent space, identity and environment are not disentangled; a pair of images of the same person in different poses/clothing can produce differences that are not purely environmental. If identity directions are present in the stored difference matrices, the orthogonality penalty in Eq. (5) will suppress identity information. The paper asserts that identity components 'largely offset each other' but provides no measurement. I request an empirical test (e.g., projecting same-concept vs cross-concept prompt differences, or ablating CVM on identity-preservation metrics) to validate this critical assump
  3. [Appendix C / §4.1] The baseline adaptation is described ambiguously and possibly unfairly. Appendix C states: 'For baselines such as MyVLM, Yo’LLaVA, and MC-LLaVA, which expand the LVLM vocabulary..., we construct corresponding training data based on the test concepts so that these methods can operate effectively.' If this training data includes the held-out query images or gold annotations, the baselines are not evaluated OOD and the SOTA claim is inflated. The main text (Sec. 4.1) says baselines were retrained using 'in-distribution data,' which conflicts with the appendix wording. Please specify exactly which images/annotations were used for each baseline, ensure no test-query leakage, and release the splits so the comparison can be reproduced.
  4. [Tables 1–2] All headline results are single-run point estimates with no error bars or significance tests. Several differences over the best baseline are small (e.g., 0.023 on OVQA single, Table 1; 0.009 on Qwen3VL captioning with β=0.6 in Table 4). The claim of 'consistently state-of-the-art' is therefore not statistically supported. The authors should report mean±std across at least three seeds for ICPT and the main baselines, or otherwise justify why single runs are sufficient. At minimum, the variance of the primary LLaVA-NeXT-7B comparison should be reported.
minor comments (5)
  1. [Tables 1–2] The 'Weight' column is not defined. Specify the weighting, presumably based on the 1050 single-concept and 2100 multi-concept test cases, and state whether the weight is the same across tasks.
  2. [Appendix B] The appendix proof of Theorem 1 is a restatement of the main-text argument rather than a self-contained proof. The claim that the parallel component is zero is an assumption of the theorem, not a derived result. Please revise to avoid circular presentation.
  3. [Table 1 / OVQA metric] Using BLEU for open-ended VQA is a weak lexical metric. Consider reporting also CIDEr or a reference-free/model-based metric to strengthen the evaluation.
  4. [Figures 4–6] The ablation figures and tables report single-point results. If multiple seeds are not available, state this explicitly in the captions.
  5. [Eq. (12)] The notation for the double sum in L_Geo is hard to parse. Clarify that the second and third terms run over j≠i, and make the order of max arguments more explicit.

Circularity Check

0 steps flagged

No significant circularity: ICPT's results come from training on held-out concepts and evaluating on unseen test concepts; the CVM/MCS regularizers are optimization constraints, not fitted predictions, and Theorem 1 is a formal algebraic bound rather than a data-derived claim.

full rationale

The paper's central claim—state-of-the-art personalization accuracy—is supported by an empirical training/evaluation pipeline, not by a derivation that reduces to its inputs. Training concepts (350) and test concepts (200) are explicitly disjoint: test concepts are 'verified to be absent from the training set' (Sec. 4.1, Appendix A.2). The CVM regularizer (Eqs. 4-5) uses pairwise differences of a concept's own reference-image prompts as a penalty direction during training; it is not fitted to test labels and does not generate a 'prediction' from a fitted parameter. Theorem 1 is mathematically correct but tautological: once δ_state is split into δ∥ ∈ span(M_ext) and δ⊥, enforcing orthogonality removes δ∥ and Cauchy-Schwarz bounds δ⊥. This is an algebraic identity implied by the definitions, not a fitted empirical result, so it does not constitute circularity; at most it shows the theorem is a weak/vacuous support for the environmental-decoupling prose claim, which is a soundness/validation concern rather than a circular one. The self-citations ([12]-[14]) motivate ICL instability and token overhead but are not load-bearing for the personalization mechanism itself, and the paper even reports failure cases (Appendix E.1) for visually similar concepts, indicating the empirical results are not forced by construction. Therefore no circular step meeting the required evidence standard is present.

Axiom & Free-Parameter Ledger

9 free parameters · 4 axioms · 2 invented entities

The method rests on several untested assumptions about reference-image structure and the transfer of soft training constraints to hard inference constraints; the main empirical claim additionally depends on a private benchmark that cannot be audited.

free parameters (9)
  • L_max (full prompt capacity) = 20
    Maximum number of visual tokens per concept; ablation in Fig. 5 shows performance peaks near 20; fixed across backbones.
  • DTR decision threshold τ = 0.5
    Pruning threshold; ablation (Fig. 7) shows inverted-U with peak at 0.5.
  • MCS margin m = 0.15
    Similarity bound between concepts; ablation (Fig. 6) shows m=0 harms performance; 0.15 chosen.
  • CVM capacity K = 128
    Queue size; ablation (Fig. 8) shows optimal K depends on batch size; K=128 at BS=8.
  • α (L_Geo weight) = 0.3 (7B), 0.6 (34B), 0.5 (InternVL3), 0.6 (Qwen3VL)
    Model-specific loss coefficient; ablation Table 4 shows moderate sensitivity.
  • β (L_Spa weight) = 0.5 (7B), 0.5 (34B), 0.4 (InternVL3), 0.4 (Qwen3VL)
    Model-specific loss coefficient; ablation Table 4.
  • Vision encoder layers used = third and third-to-last
    Feature extraction layers; ablation Table 3 shows alternatives score lower.
  • batch size = 8
    Used for all training; CVM capacity is coupled to it.
  • learning rate = 1e-4
    AdamW optimizer.
axioms (4)
  • domain assumption Reference images of the same concept are identity-stable, so pairwise prompt differences encode only environmental variation.
    Sec. 3.4, Eq. (4): CVM stores P_full_i,x - P_full_i,y as 'environment-induced variations.' If pose, viewpoint, or identity appearance shifts dominate the difference, the orthogonality penalty will erase identity information.
  • domain assumption The built-in vision encoder's early/deep features are aligned enough with the LLM that they can be projected into effective prompts without external segmentation or a second encoder.
    Sec. 3.3 uses frozen θ_V features directly; the paper validates empirically (Table 3), but the working assumption is not proven.
  • ad hoc to paper Training with soft squared-projection penalties is a sufficient proxy for the exact orthogonality condition in Theorem 1.
    Theorem 1 requires ⟨P_i, d⟩=0 exactly; L_Geo only minimizes squared projections, so the bound does not strictly hold at inference.
  • domain assumption Test concepts are truly out-of-distribution relative to training.
    Appendix A says concepts are 'verified to be absent', but the verification procedure is not specified.
invented entities (2)
  • Contextual Variation Memory (M_ext) no independent evidence
    purpose: FIFO queue of normalized pairwise prompt differences intended to span the environmental-variation subspace.
    It is an internal learned memory; its usefulness is only demonstrated through ablation on the authors' private benchmark.
  • Dynamic Token Router scores s_i,l no independent evidence
    purpose: Per-token importance scores that determine prompt length at inference.
    Internal gating mechanism; evaluated only via final task accuracy and latency.

pith-pipeline@v1.3.0-alltime-deepseek · 21732 in / 15922 out tokens · 156175 ms · 2026-08-02T12:43:10.567027+00:00 · methodology

0 comments
read the original abstract

Large vision-language models (LVLMs) have demonstrated strong general multimodal capability and are increasingly deployed in downstream systems. This trend has driven growing interest in LVLM personalization, which aims to enable models to quickly and effectively learn out-of-distribution multimodal concepts to meet user-specific needs. However, many existing methods rely on inference-time training, which reduces efficiency. They also struggle to maintain accuracy in complex multi-image, multi-concept settings. These limitations restrict the broader deployment of LVLM-based systems. Therefore, this paper proposes in-context prompt tuning (ICPT). Specifically, ICPT employs a lightweight projection module capable of operating in complex scenarios to extract fine-grained visual semantics from multiple reference images, seamlessly transforming these features alongside identity-label mappings into continuous prompts. To maximize computational efficiency, this module adaptively determines the prompt length based on the intrinsic visual complexity of each concept. Crucially, to overcome the environmental biases and cross-concept interference prevalent in real-world applications, we introduce two novel geometric regularizations. These constraints refine prompt representations by decoupling key identities from transient environmental states and separating concepts to avoid semantic confusion. Extensive experiments show that ICPT achieves state-of-the-art personalization accuracy across diverse tasks and LVLM backbones.

Figures

Figures reproduced from arXiv: 2605.31513 by Dongfang Liu, Jiaqian Li, Kuai Yu, Ruixiang Tang, Tianyang Wang, Xi Xiao, Yanshu Li.

Figure 1
Figure 1. Figure 1: Overview of LVLM personalization. Existing methods often rely on vocabulary expansion with inference-time training and struggle in multi-image and multi-concept settings. Our proposed ICPT improves efficiency and performance in complex scenarios. As these systems, which are built upon general-purpose LVLMs, naturally shift toward private and user-facing deployments, personalization emerges as a critical re… view at source ↗
Figure 2
Figure 2. Figure 2: The overall framework of ICPT. The components highlighted in pink constitute the core Adaptive Concept Projector. Separation, which mitigates cross-concept confusion while preserving shared se￾mantic structure (Sec. 3.5). Finally, we describe the training strategy (Sec. 3.6). 3.2 Multi-concept In-context Prompts For each concept Ci , an Adaptive Concept Projector (ACP) processes the ref￾erence images to ge… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative examples of ICPT on four tasks: (a) captioning, (b) open-ended VQA without a query image, (c) existence recognition, and (d) multiple VQA. More examples are provided in the Appendix [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ablation study on the DTR mechanism. Reco MVQA OVQA Captioning 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 Score m = 0.15 prompt only (m = 0.15) label only (m = 0.15) m = 0 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 8
Figure 8. Figure 8: Ablation study on CVM capacity K. visual reasoning over the query image together with an understanding of the personalized concepts. Each case involves multiple concepts, and each concept is associated with a varying number of reference images. The results show that, with ICPT, the model is able to robustly integrate user-specific concepts into general multimodal reasoning. This capability enables the LVLM… view at source ↗
Figure 9
Figure 9. Figure 9: Two additional qualitative examples of ICPT on four personalization tasks. 250 300 350 400 450 Number of training concept 0.66 0.68 0.70 0.72 0.74 0.76 0.78 0.80 Score Low Diversity Medium Diversity High Diversity [PITH_FULL_IMAGE:figures/full_fig_p026_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: ICPT’s weighted average perfor￾mance across the four task types under different training data recipes [PITH_FULL_IMAGE:figures/full_fig_p026_10.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation

    cs.CV 2026-07 conditional novelty 6.0

    PIB regularizes prompt-tuned frozen ViTs with layer-wise compression and sufficiency losses, improving VTAB-1k transfer to 77.33% with about 0.35-0.51% trainable parameters.

  2. Toward User-Conditioned Evaluation of Personal LLM Agents under Temporal Interventions

    cs.LG 2026-07 conditional novelty 6.0

    No existing agent benchmark protocol replays a fixed temporal change across varied persistent user states while measuring cross-component failure propagation, creating a concrete design gap for personal-agent evaluation.

Reference graph

Works this paper leans on

35 extracted references · 13 linked inside Pith · cited by 2 Pith papers

  1. [1]

    In: European Conference on Computer Vision

    Alaluf, Y., Richardson, E., Tulyakov, S., Aberman, K., Cohen-Or, D.: Myvlm: Personalizing vlms for user-specific queries. In: European Conference on Computer Vision. pp. 73–91. Springer (2024) 2, 4, 11, 17, 20

    2024

  2. [2]

    arXiv preprint arXiv:2411.11706 (2024) 2, 4, 11, 17, 20, 21

    An, R., Yang, S., Lu, M., Zhang, R., Zeng, K., Luo, Y., Cao, J., Liang, H., Chen, Y., She, Q., et al.: Mc-llava: Multi-concept personalized vision-language model. arXiv preprint arXiv:2411.11706 (2024) 2, 4, 11, 17, 20, 21

    arXiv 2024

  3. [3]

    arXiv preprint arXiv:2505.14671 (2025) 4

    An, R., Yang, S., Zhang, R., Shen, Z., Lu, M., Dai, G., Liang, H., Guo, Z., Yan, S., Luo, Y., et al.: Unictokens: Boosting personalized understanding and generation via unified concept tokens. arXiv preprint arXiv:2505.14671 (2025) 4

    arXiv 2025

  4. [4]

    In: Synthetic Data for Computer Vision Workshop@ CVPR 2025 (2025) 4

    An, R., Zeng, K., Lu, M., Yang, S., Zhang, R., Ji, H., Zhang, Q., Luo, Y., Liang, H., Zhang, W.: Concept-as-tree: Synthetic data is all you need for vlm personalization. In: Synthetic Data for Computer Vision Workshop@ CVPR 2025 (2025) 4

    2025

  5. [5]

    arXiv preprint arXiv:2511.21631 (2025) 4, 11

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng, L., Ren, X., Ren, X., Song, S., Sun, Y., Tang, ...

    Pith/arXiv arXiv 2025

  6. [6]

    arXiv preprint arXiv:2306.08640 (2023) 4

    Gao, D., Ji, L., Zhou, L., Lin, K.Q., Chen, J., Fan, Z., Shou, M.Z.: Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn. arXiv preprint arXiv:2306.08640 (2023) 4

    Pith/arXiv arXiv 2023

  7. [7]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Hao, H., Han, J., Li, C., Li, Y.F., Yue, X.: Rap: Retrieval-augmented personal- ization for multimodal large language models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 14538–14548 (2025) 4, 11, 21 16 F. Author et al

    2025

  8. [8]

    arXiv preprint arXiv:2111.09543 (2021) 12

    He, P., Gao, J., Chen, W.: Debertav3: Improving deberta using electra- style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543 (2021) 12

    Pith/arXiv arXiv 2021

  9. [9]

    arXiv preprint arXiv:2509.22820 (2025) 4, 10, 19

    Kim, J., Kim, W., Park, W., Do, J.: Mmpb: It’s time for multi-modal personaliza- tion. arXiv preprint arXiv:2509.22820 (2025) 4, 10, 19

    arXiv 2025

  10. [10]

    arXiv preprint arXiv:2509.21730 (2025) 4

    Kim, J., Choi, J., Chay, W., Kyung, D., Kwon, Y., Jo, Y., Choi, E.: Propersim: De- veloping proactive and personalized ai assistants through user-assistant simulation. arXiv preprint arXiv:2509.21730 (2025) 4

    arXiv 2025

  11. [11]

    International Journal of Human–Computer Interaction pp

    Li, F., Han, S., Lee, C.H., Feng, S., Jiang, Z., Sun, Z.: A new era in human factors engineering:Asurveyoftheapplicationsandprospectsoflargemultimodalmodels. International Journal of Human–Computer Interaction pp. 1–14 (2025) 1

    2025

  12. [12]

    In: Second Conference on Language Modeling (2025), https://openreview.net/forum?id=9ffYcEiNw92

    Li, Y., Cao, Y., He, H., Cheng, Q., Fu, X., Xiao, X., Wang, T., Tang, R.: M²IV: Towards efficient and fine-grained multimodal in-context learning via rep- resentation engineering. In: Second Conference on Language Modeling (2025), https://openreview.net/forum?id=9ffYcEiNw92

    2025

  13. [13]

    In: Pro- ceedings of the AAAI Conference on Artificial Intelligence

    Li, Y., Yang, J., Shen, Z., Han, L., Xu, H., Tang, R.: Catp: Contextually adaptive token pruning for efficient and enhanced multimodal in-context learning. In: Pro- ceedings of the AAAI Conference on Artificial Intelligence. vol. 40, pp. 6619–6627 (2026) 2

    2026

  14. [14]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Li, Y., Yang, J., Yang, Z., Li, B., Han, L., He, H., Yao, Z., Chen, Y.V., Fei, S., Liu, D., et al.: Make lvlms focus: Context-aware attention modulation for better mul- timodal in-context learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 40, pp. 6610–6618 (2026) 2

    2026

  15. [15]

    In: European conference on computer vision

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014) 10

    2014

  16. [16]

    Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., Lee, Y.J.: Llava-next: Improved reasoning, ocr, and world knowledge (January 2024) 11

    2024

  17. [17]

    Advances in neural information processing systems36, 34892–34916 (2023) 4

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36, 34892–34916 (2023) 4

    2023

  18. [18]

    In: Findings of the Association for Computational Linguistics: NAACL 2024

    Lyu, H., Jiang, S., Zeng, H., Xia, Y., Wang, Q., Zhang, S., Chen, R., Leung, C., Tang, J., Luo, J.: Llm-rec: Personalized recommendation via prompting large language models. In: Findings of the Association for Computational Linguistics: NAACL 2024. pp. 583–612 (2024) 2

    2024

  19. [19]

    Ad- vances in Neural Information Processing Systems37, 23464–23487 (2024) 6

    Meng, L., Yang, J., Tian, R., Dai, X., Wu, Z., Gao, J., Jiang, Y.G.: Deepstack: Deeply stacking visual tokens is surprisingly simple and effective for lmms. Ad- vances in Neural Information Processing Systems37, 23464–23487 (2024) 6

    2024

  20. [20]

    Advances in Neural Information Processing Systems 37, 40913–40951 (2024) 2, 4, 11, 17, 20

    Nguyen,T.,Liu,H.,Li,Y.,Cai,M.,Ojha,U.,Lee,Y.J.:Yo’llava:Yourpersonalized language and vision assistant. Advances in Neural Information Processing Systems 37, 40913–40951 (2024) 2, 4, 11, 17, 20

    2024

  21. [21]

    arXiv preprint arXiv:2304.07193 (2023) 22

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023) 22

    Pith/arXiv arXiv 2023

  22. [22]

    In: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems

    Park, S., Song, Y., Lee, S., Kim, J., Seo, J.: Leveraging multimodal llm for inspira- tional user interface search. In: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. pp. 1–22 (2025) 4

    2025

  23. [23]

    arXiv preprint arXiv:2412.17610 (2024) 2, 4, 11, 21

    Pham, C., Phan, H., Doermann, D., Tian, Y.: Personalized large vision-language models. arXiv preprint arXiv:2412.17610 (2024) 2, 4, 11, 21

    Pith/arXiv arXiv 2024

  24. [24]

    arXiv preprint arXiv:2410.07113 (2024) 2, 4, 11, 21 Abbreviated paper title 17

    Pi, R., Zhang, J., Han, T., Zhang, J., Pan, R., Zhang, T.: Personalized visual instruction tuning. arXiv preprint arXiv:2410.07113 (2024) 2, 4, 11, 21 Abbreviated paper title 17

    Pith/arXiv arXiv 2024

  25. [25]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021) 3

    2021

  26. [26]

    arXiv preprint arXiv:2401.14159 (2024) 22

    Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., et al.: Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 (2024) 22

    Pith/arXiv arXiv 2024

  27. [27]

    In: Proceedings of the 62nd Annual Meeting of the AssociationforComputationalLinguistics(Volume1:LongPapers).pp.7370–7392 (2024) 4

    Salemi, A., Mysore, S., Bendersky, M., Zamani, H.: Lamp: When large language models meet personalization. In: Proceedings of the 62nd Annual Meeting of the AssociationforComputationalLinguistics(Volume1:LongPapers).pp.7370–7392 (2024) 4

    2024

  28. [28]

    arXiv preprint arXiv:2502.02452 (2025) 11, 21

    Seifi, S., Dorovatas, V., Reino, D.O., Aljundi, R.: Personalization toolkit: Training free personalization of large vision language models. arXiv preprint arXiv:2502.02452 (2025) 11, 21

    Pith/arXiv arXiv 2025

  29. [29]

    arXiv preprint arXiv:2508.10104 (2025) 22

    Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khali- dov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., et al.: Dinov3. arXiv preprint arXiv:2508.10104 (2025) 22

    Pith/arXiv arXiv 2025

  30. [30]

    arXiv preprint arXiv:2505.19897 (2025) 1

    Sun, Q., Liu, Z., Ma, C., Ding, Z., Xu, F., Yin, Z., Zhao, H., Wu, Z., Cheng, K., Liu, Z., et al.: Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows. arXiv preprint arXiv:2505.19897 (2025) 1

    Pith/arXiv arXiv 2025

  31. [31]

    5: Advancing open-source multimodal models in versatility, reasoning, and efficiency

    Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al.: Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265 (2025) 4

    Pith/arXiv arXiv 2025

  32. [32]

    arXiv preprint arXiv:2510.22765 (2025) 4

    Xu, B., Feng, J., Lu, S., Luo, Y., Yan, S., Liang, H., Lu, M., Zhang, W.: Jarvis: Towards personalized ai assistant via personal kv-cache retrieval. arXiv preprint arXiv:2510.22765 (2025) 4

    arXiv 2025

  33. [33]

    National Science Review11(12), nwae403 (2024) 3

    Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., Chen, E.: A survey on multimodal large language models. National Science Review11(12), nwae403 (2024) 3

    2024

  34. [34]

    arXiv preprint arXiv:2401.13601 (2024) 1, 4

    Zhang, D., Yu, Y., Dong, J., Li, C., Su, D., Chu, C., Yu, D.: Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601 (2024) 1, 4

    Pith/arXiv arXiv 2024

  35. [35]

    Can you see <sks1> in this photo? Answer with a single word: Yes or No

    Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., et al.: Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 (2025) 11 A Personalization Data A.1 Training data When constructing the multi-image, multi-concept personalization dataset, we f...

    Pith/arXiv arXiv 2025