Group Preference Collapse in Personalized Multimodal Large Language Models

Fan Lyu; Joost Van De Weijer; Wenqi Zhang

REVIEW 2 major objections 4 minor 46 references

Group Preference Collapse in Personalized Multimodal Large Language Models

T0 review · 2 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Personalized multimodal models can preserve individual tastes if preferences are learned as residuals over shared prototypes, rather than compressed into a single user signal.

desk verdict Names a plausible failure mode and a coherent fix, but the headline numbers rest on a reconstruction rule the authors control, so the collapse reduction is plausible but unverified. read the letter →

arxiv 2607.22603 v1 pith:XAPWZHKY submitted 2026-06-12 cs.AI

Fan Lyu , Wenqi Zhang , Joost van de Weijer This is my paper

classification cs.AI

keywords grouppreferencecollapsepersonalizedmultimodalLLMlearningprototype-residualdecompositionmixture-of-expertsroutingLoRAadaptationimbalance-awarecontrastive

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Personalized multimodal large language models are supposed to answer in ways that match the individual user, but the paper finds they often collapse to the majority answer even when explicit user preferences are injected. The cause is two-fold: rare preferences get absorbed into dominant patterns during representation learning, and the model fails to use the right preference cue during generation. The paper proposes PrefMoE, which keeps profile and preference information separate, writes each user's preference as a shared prototype plus a personal residual, and protects minority residuals with reweighted contrastive learning, counterfactual pseudo-users, and decorrelation. A hierarchical mixture-of-experts router then lets the model read off profile or preference info only when relevant. Across several backbones PrefMoE lifts preference accuracy and roughly halves the collapse rate relative to full fine-tuning.

What carries the argument

The load-bearing device is the 'prototype-plus-residual' preference factorization combined with a hierarchical LoRA router. Each user's preference on a facet is written as a shared prototype plus a personalized residual; the residual is what distinguishes one user from the crowd. The framework preserves these residuals through an imbalance-aware contrastive loss that up-weights low-frequency groups, a counterfactual pseudo-user augmentation that breaks group-level shortcuts, and a decorrelation penalty that keeps residual directions from being redundant across facets or absorbed into the prototype. A two-level router then activates facet-specific and profile-specific LoRA experts depending o

What would settle it

On an independent benchmark with human-annotated per-user preference labels (not reconstructed from shared semantics), compare PrefMoE against full fine-tuning on preference accuracy and collapse. If the gap vanishes or reverses, the reported collapse reduction is an artifact of the label-construction protocol rather than of the factorization. Also, the headline collapse numbers are based on only 219 boundary-external samples; bootstrapping the evaluation set would show whether the 34.25% to 12.33% improvement is within sampling noise.

Watch

Extended reading notes

Core claim

The central claim is that 'group preference collapse' is a structural failure of personalized MLLMs, not a mere data issue. When many users are trained jointly, low-frequency preferences are suppressed by population-level patterns, and even injected preference cues go unused because the model follows visual priors or frequent responses. PrefMoE counters this by factorizing the user representation: a profile branch (image and description embeddings) and a preference branch where each of five facets is decomposed into a shared prototype and an individualized residual. The residuals are preserved by an imbalance-aware focal-style contrastive loss over query-conditioned activations, by augmentin

Load-bearing premise

The premise that the MMPB-Clean reconstruction — where an image group's semantic meaning is read off the shared preferences of its associated users and negative pairs are drawn from users whose preference fields lack that meaning — is a neutral ruler for measuring collapse.

Editorial extensions

If this is right

Preference-conditioned MLLMs can answer per-user rather than per-crowd, provided preference and profile signals are kept in separate channels.
Minority and low-frequency preferences are preservable without re-balancing data, as long as the residual space is explicitly contrasted and decorrelated.
Profile-recognition accuracy does not have to suffer when preference reasoning is added; in the reported results both improve together.
The method works across multiple backbones and under 10-turn dialogue history, so the factorization is not tuned to one architecture.
The collapse metric, false positives on boundary-external items, is reduced in every preference-popularity bucket, meaning the effect is not only about frequent items.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If group preference collapse generalizes to text-only personalized LLMs, the same prototype-plus-residual decomposition should reduce majority bias in recommender generation; this is a direct testable extension the paper does not run.
The evaluation's label reconstruction ('image-semantics to user-preference matching') shares structure with the model's facet decomposition, so part of the measured collapse reduction could be an artifact of how the test set is built; an independent benchmark with human-collected labels would settle this.
Because the method needs explicit facet-structured preference descriptions, its gains likely shrink when preferences are underspecified or not visually grounded — a limitation the paper itself acknowledges.
The failure mode is analogous to popularity bias in recommendation systems, so the framework may connect to fairness work on under-served user groups, though the paper does not discuss this.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this paper is worth a serious referee, but the central quantitative claims are not yet believable as stated. The phenomenon it names — group preference collapse in multi-user personalized MLLMs — is real and easy to demonstrate on raw models: no-training collapse is 50–70% across backbones, and FFT only pulls part of it down. The proposed mechanism (prototype + residual decomposition, imbalance-aware contrastive with counterfactual pseudo-users, decorrelation, factorized LoRA routing) is a reasonable synthesis and the combination is new as far as the cited literature goes. The authors also do a non-trivial thing: they report failure cases and limitations honestly.

The soft spot is the evaluation. MMPB-Clean is an author-constructed split where preference labels are reconstructed by checking whether the image group's semantic meaning appears in the user's preference text. That operationalizes 'not stated in the preference text' as 'does not prefer.' Since PrefMoE is trained with a VQA loss on those very labels plus auxiliary losses using the same facet annotations, the evaluation risks measuring how well the model learns the text-containment rule rather than genuine preference preservation. The collapse metric also runs on only 219 boundary-external samples, so the 34.25%→12.33% claim has sampling uncertainty before any run-to-run variance. No error bars, seeds, or significance tests anywhere. Code and split files are promised but not released. These are all fixable.

My own read: the paper is internally coherent and the math checks out — the micro-averaging in Table 1 is consistent. The core mechanism is plausible. But the stress-test concern lands: the measurement instrument is not independent of the method. I don't think the authors are being sneaky; the protocol is described in detail and applied to all methods equally. It just means the results are conditional on accepting MMPB-Clean as a neutral ruler, and I can't fully accept that without artifacts and a calibration against the original MMPB protocol.

Who should read it: anyone working on personalized MLLMs or preference-conditioned generation. It names a failure mode worth knowing about even if you disagree with the numbers.

Recommendation: send it to peer review, but require release of the split and code, multi-seed variance, and a comparison against the original MMPB protocol before accepting. The reader's CONDITIONAL verdict is about right; I'd be slightly more skeptical on the benchmark validity, but that's a reason for revision, not rejection.

Referee Report

2 major / 4 minor

Summary. The paper identifies a failure mode it calls group preference collapse in multi-user personalized MLLMs: models drift toward population-dominant preferences and ignore individual preference signals. It proposes PrefMoE, which factorizes user representation into profile factors and facet-wise preference factors, decomposes each preference factor into a shared prototype and a personalized residual, and regularizes residuals with imbalance-aware contrastive learning, counterfactual pseudo-user augmentation, and residual decorrelation. A hierarchical MoE router with separate profile and preference LoRA experts is used to turn these factors into query-dependent reasoning paths. The main empirical claim is that PrefMoE substantially improves preference-sensitive accuracy and reduces collapse across LLaVA, DeepSeek-VL2, and Qwen2.5-VL backbones on the author-constructed MMPB-Clean evaluation split, with headline 0-turn LLaVA-1.5-7B improvements from 44.13% to 67.33% preference accuracy and collapse reduction from 34.25% to 12.33% over full fine-tuning.

Significance. If valid, the paper makes a useful conceptual and practical contribution: it names a measurable failure mode, provides an implementable training recipe, and evaluates across multiple backbones. The formal collapse definition in Appendix C, the dataset-construction detail in Appendix B, and the component ablation in Table 2 are valuable and go beyond what is typical in a short conference submission. However, the central evaluation rests on the validity of MMPB-Clean as an independent measuring instrument, and the boundary-external set used for the collapse metric contains only 219 samples. The method's internal math appears coherent and the equations are implementable, but the empirical evidence as presented does not yet establish that the measured collapse reduction corresponds to genuine preservation of user-specific preference boundaries.

major comments (2)

[Appendix B.2; Eq. (13); Appendix C] The central empirical claim is supported by a measurement instrument that is not independent of the method. MMPB-Clean reconstructs preference labels by defining an image group's semantic meaning as 'stable preference elements shared by the associated users' and labeling a query positive iff the element appears in the target user's preference field, and negative otherwise. PrefMoE is trained with L_vqa on exactly these reconstructed labels (Eq. 13) and with L_pref/L_res using the same facet annotations (Eqs. 5-7), so the training signal and the collapse metric (false-positive rate on the boundary-external set O, App. C) encode the same text-containment rule. The statement in App. B.3 that the labels are 'independent of our model design' does not address this: the reconstruction is organized around the same facet-aligned preference structure that PrefMoE is deliberately built to exploit.
[App. D.1; Table 1] The collapse metric is computed on only 219 boundary-external samples in total (n<=4: 50, n5-8: 121, n>=9: 48). The headline 0-turn LLaVA-1.5-7B reduction from 0.3425 to 0.1233 corresponds to about 75 versus 27 positive predictions in O, i.e., a difference of roughly 48 binary labels. No standard errors, confidence intervals, or multiple-seed results are reported anywhere, so it is unclear whether small cross-method gaps in Table 1 (e.g., 0.1096 vs 0.1142 vs 0.1233) are meaningful. Report binomial confidence intervals and at least three seeds, and preferably also per-popularity-bucket intervals, before making strong comparative claims about collapse reduction.

minor comments (4)

[Table 1] Entries are run together in the printed table (e.g., 'LLaV A-1.5-7BNT0.3564' should be 'LLaVA-1.5-7B NT 0.3564'). This makes the table very hard to parse; please fix spacing and alignment.
[Fig. 4] The figure is too small and the axes/legend are essentially unreadable; the 'w/ facet label vs w/o facet label' comparison is important but currently can only be interpreted visually. Provide a numeric table and error bars.
[Eqs. (5)-(6)] The role of counterfactual pseudo-users in the contrastive objective is underspecified. Since pseudo-users are added to the candidate set B, it should be stated explicitly whether they also enter the positive set Ni,f, and how their facet-level preference annotations are defined for that purpose.
[App. B.3] The appendix promises release of split files, image-group identifiers, label files, and evaluation scripts, but no URL, repository, or DOI is given. The current claims are therefore not independently checkable; please provide the actual artifact or an anonymous link.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivation-level circularity; results are empirical and the evaluation-split concerns are external validity, not equation-level reduction.

full rationale

This paper makes empirical claims measured on a benchmark; there is no derivation chain in which a predicted quantity reduces by construction to an input. The training objective (Eq. 13) is a sum of VQA, profile, and preference losses, and the reported headline numbers are measured accuracies and collapse rates, not quantities derived from the model definition. MMPB-Clean is an author-constructed evaluation split, and one could question whether its reconstructed preference labels are a valid ground truth, but the paper explicitly states that all methods use the same reconstructed labels and that the reconstruction is independent of the factorized user representation, residual learning, and MoE routing. Such label-validity concerns affect interpretation and external generalizability, not circularity in the sense of Eq. X = Eq. Y by construction or a fitted parameter being renamed a prediction. Self-citations in the reference list (e.g., [23], [36], [38]) are related-work citations and are not load-bearing for the central claims; no uniqueness theorem from the authors is invoked, no ansatz is imported via self-citation, and no known empirical pattern is merely renamed as a derivation. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

This paper's contribution is empirical, not derivational, so the ledger mostly records hyperparameters, benchmark-construction choices, and modeling assumptions rather than fitted constants. The two structural bet-the-paper choices are the additive prototype+residual decomposition and the self-constructed MMPB-Clean labels; both are unverified assumptions about how preferences are actually structured.

free parameters (5)

number of counterfactual pseudo-users = 50
Chosen via the Fig. 3 ablation ('performance improves... from 10 to 50, while both Preference and Collapse become saturated afterward'); a tuned hyperparameter, not derived.
imbalance/focal exponent η
Controls minority reweighting strength in Eq. (6); no value is reported anywhere in the paper.
contrastive temperature τ
Appears in Eq. (5) for residual contrast; no value or sensitivity analysis reported.
loss weighting of Lvqa, Lprof, Lpref = 1 : 1 : 1
Eq. (13) sums the three objectives with implicit unit weights; no balancing or sensitivity analysis reported.
LoRA capacity (rank 64, top 8 layers, 16 insertion sites) = rank-64, 16 sites
Architecture capacity asserted in App. A without ablation; the number of sites scales with decoder depth for other backbones.

assumptions (5)

domain assumption User preferences decompose additively into a shared prototype per facet plus a personalized residual (Eq. 4: z = z̄ + Δ).
The entire residual-learning machinery presumes this additive structure is adequate to capture preference diversity; no evidence for the decomposition's adequacy is provided.
domain assumption MMPB's five-facet preference annotations (entertainment, travel, lifestyle, shopping, fashion) faithfully represent user preferences.
These annotations are simultaneously the training target, the contrastive supervision (Eq. 5), and the evaluation ground truth (App. C); their validity is inherited from the benchmark.
ad hoc to paper MMPB-Clean label reconstruction is valid: image-group semantic meaning can be extracted as 'stable preference elements shared by the associated users,' and users lacking those elements are valid negative labels.
This is a new protocol created by the authors (App. B.2); the central measured quantities (accuracy, collapse) are defined on these reconstructed labels.
domain assumption Query-dependent softmax routing over facet experts (Eqs. 9–12) has sufficient capacity and stability to select relevant preferences per question.
The router is trained end-to-end with the VQA loss; no analysis of routing collapse (all queries picking the same facets) is provided, although Fig. 5 partially addresses branch-level routing.
domain assumption PEFT/LoRA adapters in top decoder layers preserve backbone competence while adding personalization.
Standard LoRA assumption inherited from prior PEFT literature; the paper provides no evidence that inserted adapters do not degrade general capability.

invented entities (2)

Group preference collapse (named failure mode)
purpose: Frames the problem: models drift toward dominant population-level preferences even with explicit user cues; measured as FPR on boundary-external items (App. C).
The construct is measured only through the authors' own reconstructed labels and their own FPR definition; no external benchmark or independent dataset establishes the phenomenon's prevalence.
Personalized residual Δi,f (and shared prototype z̄f)
purpose: Represents what is individual to user i within facet f, separate from the population tendency; trained via contrastive loss, augmentation, and decorrelation.
Internal representational machinery with no independent empirical handle; it is a modeling device rather than an empirically isolated quantity (in contrast to a predicted particle mass or a falsifiable constant).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Group Preference Collapse in Personalized Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/XAPWZHKY

@misc{pith2026260722603,
  author       = {Pith},
  title        = {Pith review of: Group Preference Collapse in Personalized Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XAPWZHKY}},
  note         = {Machine review of arXiv:2607.22603}
}

read the original abstract

Personalized multimodal large language models (MLLMs) aim to generate user-specific responses, but existing methods mainly rely on profile-level information and overlook diverse user preferences. We identify group preference collapse, where multi-user personalized MLLMs become insensitive to individual preferences and drift toward dominant population-level choices due to suppressed preference signals and unreliable preference use during generation. We propose PrefMoE, a preference-centric framework that separates stable profile information from preference-related representations. PrefMoE decomposes preferences into shared prototypes and personalized residuals, preserves individualized residuals with imbalance-aware learning, counterfactual pseudo-user augmentation, and residual decorrelation, and routes profile and preference factors through separate LoRA adaptation paths. Experiments across multiple MLLM backbones show that PrefMoE improves preference-sensitive personalization while substantially reducing preference collapse. Project page: https://prefmoe.github.io/.

Figures

Figures reproduced from arXiv: 2607.22603 by the authors.

**Figure 1.** Illustration of group preference collapse. Although users have diverse preferences, existing personalized MLLMs tend to predict the dominant preference across users (“Yoga”) rather than reflecting the target user’s specific preference. Bars indicate the percentage of users from each ground-truth preference group whose Top-1 preference is the dominant preference (“Yoga”). We attribute group preference collapse to two… view at source ↗

**Figure 2.** Overview of PrefMoE. PrefMoE separates profile and preference representations, mod [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 5.** Routing analysis. Adaptive routing vs. fixed routing [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

Figures from the paper (2 more)

**Figure 7.** Figure 7: t-SNE visualization of top-1 to top-4 preference groups within [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

**Figure 8.** Figure 8: Qualitative examples. <SKS> denotes the target personalized identity or concept. Green boxes indicate correct predictions, while red marks indicate incorrect predictions. target-agnostic descriptions by focusing on salient visual content, while PrefMoE better condition…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 5 linked inside Pith

[1]

MyVLM: Personalizing VLMs for user-specific queries

Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, and Daniel Cohen-Or. MyVLM: Personalizing VLMs for user-specific queries. InECCV, pages 73–91, 2024

2024
[2]

Flamingo: A visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Ruther- ford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Miko- laj Binkow...

2022
[3]

MC-LLaV A: Multi-concept personalized vision-language model.arXiv preprint arXiv:2411.11706, 2024

Ruichuan An, Sihan Yang, Ming Lu, Kai Zeng, Yulin Luo, Ying Chen, Jiajun Cao, Hao Liang, Qi She, Shanghang Zhang, and Wentao Zhang. MC-LLaV A: Multi-concept personalized vision-language model.arXiv preprint arXiv:2411.11706, 2024

arXiv 2024
[4]

Lawrence Zitnick, and Devi Parikh

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InICCV, December 2015

2015
[5]

Online-PVLM: Advancing personalized VLMs with online concept learning.arXiv preprint arXiv:2511.20056, 2025

Huiyu Bai, Runze Wang, Zhuoyun Du, Yiyang Zhao, Fengji Zhang, Haoyu Chen, Xiaoyong Zhu, Bo Zheng, and Xuejiao Zhao. Online-PVLM: Advancing personalized VLMs with online concept learning.arXiv preprint arXiv:2511.20056, 2025

arXiv 2025
[6]

Qwen2.5-VL technical report

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL technical report. a...

arXiv 2025
[7]

Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

arXiv 2022
[8]

Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3–4):324–345, 1952

1952

Show all 46 references

[9]

Speaker role contextual modeling for language understanding and dialogue policy learning

Ta-Chung Chi, Po-Chun Chen, Shang-Yu Su, and Yun-Nung Chen. Speaker role contextual modeling for language understanding and dialogue policy learning. InIJCNLP, pages 163–168, 2017

2017
[10]

Christiano, Jan Leike, Tom B

Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. InNeurIPS, volume 30, 2017

2017
[11]

this is my unicorn, fluffy

Niv Cohen, Rinon Gal, Eli A. Meirom, Gal Chechik, and Yuval Atzmon. “this is my unicorn, fluffy”: Personalizing frozen vision-language representations. InECCV, pages 558–577, 2022

2022
[12]

Class-balanced loss based on effective number of samples

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. InCVPR, pages 9268–9277, 2019

2019
[13]

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. InNeurIPS, volume 36, 2023

2023
[14]

KTO: Model alignment as prospect theoretic optimization

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: Model alignment as prospect theoretic optimization. InICML, volume 235, pages 12634–12651, 2024. 10

2024
[15]

Guided mllm reasoning: Enhancing mllm with knowledge and visual notes for visual question answering

Wenlong Fang, Qiaofeng Wu, Jing Chen, and Yun Xue. Guided mllm reasoning: Enhancing mllm with knowledge and visual notes for visual question answering. InCVPR, pages 19597– 19607, 2025

2025
[16]

Procaccia, Itai Shapira, Yevgeniy V orobeychik, and Junlin Wu

Luise Ge, Daniel Halpern, Evi Micha, Ariel D. Procaccia, Itai Shapira, Yevgeniy V orobeychik, and Junlin Wu. Axioms for AI alignment from human feedback. InNeurIPS, volume 37, 2024

2024
[17]

Bradley Knox, and Dorsa Sadigh

Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without reinforcement learning. InICLR, 2024

2024
[18]

Boosting mllm reasoning with text-debiased hint-grpo

Qihan Huang, Weilong Dai, Jinlong Liu, Wanggui He, Hao Jiang, Mingli Song, Jingyuan Chen, Chang Yao, and Jie Song. Boosting mllm reasoning with text-debiased hint-grpo. InICCV, pages 4848–4857, 2025

2025
[19]

MMPB: It’s time for multi-modal personalization

Jaeik Kim, Woojin Kim, Woohyeon Park, and Jaeyoung Do. MMPB: It’s time for multi-modal personalization. InNeurIPS Datasets and Benchmarks Track, 2025

2025
[20]

LLaV A-OneVision: Easy visual task transfer

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaV A-OneVision: Easy visual task transfer. TMLR, 2025

2025
[21]

Spithourakis, Jianfeng Gao, and Bill Dolan

Jiwei Li, Michel Galley, Chris Brockett, Georgios P. Spithourakis, Jianfeng Gao, and Bill Dolan. A persona-based neural conversation model. InACL, pages 994–1003, 2016

2016
[22]

Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. InICML, pages 19730–19742, 2023

2023
[23]

Mm-prompt: Cross-modal prompt tuning for continual visual question answering.arXiv preprint arXiv:2505.19455, 2025

Xu Li and Fan Lyu. Mm-prompt: Cross-modal prompt tuning for continual visual question answering.arXiv preprint arXiv:2505.19455, 2025

2025 arXiv
[24]

Focal loss for dense object detection

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. InICCV, pages 2980–2988, 2017

2017
[25]

Improved baselines with visual instruction tuning

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InCVPR, 2024

2024
[26]

LLaV A-NeXT: Improved reasoning, OCR, and world knowledge

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaV A-NeXT: Improved reasoning, OCR, and world knowledge. Technical blog, 2024

2024
[27]

Decoupled weight decay regularization

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InICLR, 2019

2019
[28]

Yo’LLaV A: Your personalized language and vision assistant

Thao Nguyen, Haotian Liu, Yuheng Li, Mu Cai, Utkarsh Ojha, and Yong Jae Lee. Yo’LLaV A: Your personalized language and vision assistant. InNeurIPS, volume 37, 2024

2024
[29]

Yo’Chameleon: Personalized vision and language generation

Thao Nguyen, Krishna Kumar Singh, Jing Shi, Trung Bui, Yong Jae Lee, and Yuheng Li. Yo’Chameleon: Personalized vision and language generation. InCVPR, pages 14438–14448, 2025

2025
[30]

Christiano, Jan Leike, and Ryan Lowe

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, ...

2022
[31]

PLVM: A tuning-free approach for personalized large vision-language model

Chau Pham, Hoang Phan, David Doermann, and Yunjie Tian. PLVM: A tuning-free approach for personalized large vision-language model. InCVPR Workshops, pages 3671–3680, 2025

2025
[32]

Personalized visual instruction tuning

Renjie Pi, Jianshu Zhang, Tianyang Han, Jipeng Zhang, Rui Pan, and Tong Zhang. Personalized visual instruction tuning. InICLR, 2025

2025
[33]

Manning, and Chelsea Finn

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InNeurIPS, volume 36, 2023. 11

2023
[34]

BPR: Bayesian personalized ranking from implicit feedback

Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. BPR: Bayesian personalized ranking from implicit feedback. InUAI, pages 452–461, 2009

2009
[35]

Personalization toolkit: Training free personalization of large vision language models.TMLR, 2026

Soroush Seifi, Vaggelis Dorovatas, Matteo Cassinelli, Fabien Despinoy, Daniel Olmeda Reino, and Rahaf Aljundi. Personalization toolkit: Training free personalization of large vision language models.TMLR, 2026

2026
[36]

Towards dynamic modality alignment in multimodal continual learning

Jiayao Tan, Fan Lyu, Tianle Liu, Fuyuan Hu, and Wei Feng. Towards dynamic modality alignment in multimodal continual learning. InCVPR, pages 39911–39921, 2026

2026
[37]

Visual position prompt for MLLM based visual grounding.IEEE TMM, 2026

Wei Tang, Yanpeng Sun, Qinying Gu, and Zechao Li. Visual position prompt for MLLM based visual grounding.IEEE TMM, 2026

2026
[38]

Partition-then-adapt: Combating prediction bias for reliable multi-modal test-time adaptation

Guowei Wang, Fan Lyu, and Changxing Ding. Partition-then-adapt: Combating prediction bias for reliable multi-modal test-time adaptation. InNeurIPS, 2025

2025
[39]

Marten: Visual question answering with mask generation for multi-modal document understanding

Zining Wang, Tongkun Guan, Pei Fu, Chen Duan, Qianyi Jiang, Zhentao Guo, Shan Guo, Junfeng Luo, Wei Shen, and Xiaokang Yang. Marten: Visual question answering with mask generation for multi-modal document understanding. InCVPR, pages 14460–14471, 2025

2025
[40]

DeepSeek-VL2: Mixture-of-experts vision-language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302, 2024

Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang,...

2024 arXiv
[41]

TG-LLaV A: Text guided LLaV A via learnable latent embeddings.Proceedings of the AAAI Conference on Artificial Intelligence, 39(9):9076–9084, 2025

Dawei Yan, Pengcheng Li, Yang Li, Hao Chen, Qingguo Chen, Weihua Luo, Wei Dong, Qingsen Yan, Haokui Zhang, and Chunhua Shen. TG-LLaV A: Text guided LLaV A via learnable latent embeddings.Proceedings of the AAAI Conference on Artificial Intelligence, 39(9):9076–9084, 2025

2025
[42]

Meta- personalizing vision-language models to find named instances in video

Chun-Hsiao Yeh, Bryan Russell, Josef Sivic, Fabian Caba Heilbron, and Simon Jenni. Meta- personalizing vision-language models to find named instances in video. InCVPR, pages 19123–19132, 2023

2023
[43]

Personalizing dialogue agents: I have a dog, do you have pets too? InACL, pages 2204–2213, 2018

Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: I have a dog, do you have pets too? InACL, pages 2204–2213, 2018

2018
[44]

LOV A3: Learning to visual question answering, asking and assessment

Henry Hengyuan Zhao, Pan Zhou, Difei Gao, Zechen Bai, and Mike Zheng Shou. LOV A3: Learning to visual question answering, asking and assessment. InNeurIPS, volume 37, 2024

2024
[45]

MiniGPT-4: Enhancing vision-language understanding with advanced large language models

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. InICLR, 2024

2024
[46]

Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. 12 Appendix A Implementation Details We instantiate our fra...

1909 arXiv

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

MyVLM: Personalizing VLMs for user-specific queries

Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, and Daniel Cohen-Or. MyVLM: Personalizing VLMs for user-specific queries. InECCV, pages 73–91, 2024

2024

[2] [2]

Flamingo: A visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Ruther- ford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Miko- laj Binkow...

2022

[3] [3]

MC-LLaV A: Multi-concept personalized vision-language model.arXiv preprint arXiv:2411.11706, 2024

Ruichuan An, Sihan Yang, Ming Lu, Kai Zeng, Yulin Luo, Ying Chen, Jiajun Cao, Hao Liang, Qi She, Shanghang Zhang, and Wentao Zhang. MC-LLaV A: Multi-concept personalized vision-language model.arXiv preprint arXiv:2411.11706, 2024

arXiv 2024

[4] [4]

Lawrence Zitnick, and Devi Parikh

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InICCV, December 2015

2015

[5] [5]

Online-PVLM: Advancing personalized VLMs with online concept learning.arXiv preprint arXiv:2511.20056, 2025

Huiyu Bai, Runze Wang, Zhuoyun Du, Yiyang Zhao, Fengji Zhang, Haoyu Chen, Xiaoyong Zhu, Bo Zheng, and Xuejiao Zhao. Online-PVLM: Advancing personalized VLMs with online concept learning.arXiv preprint arXiv:2511.20056, 2025

arXiv 2025

[6] [6]

Qwen2.5-VL technical report

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL technical report. a...

arXiv 2025

[7] [7]

Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

arXiv 2022

[8] [8]

Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3–4):324–345, 1952

1952