Pith. sign in

REVIEW 5 cited by

Yo'LLaVA: Your Personalized Language and Vision Assistant

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.09400 v2 pith:QSRJ3KTV submitted 2024-06-13 cs.CV cs.LG

Thao Nguyen , Haotian Liu , Yuheng Li , Mu Cai , Utkarsh Ojha , Yong Jae Lee This is my paper · ORCID
classification cs.CVcs.LG
keywords llavagenericpersonalizedsubjectbirthdayexamplefriendholding
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large Multimodal Models (LMMs) have shown remarkable capabilities across a variety of tasks (e.g., image captioning, visual question answering). While broad, their knowledge remains generic (e.g., recognizing a dog), and they are unable to handle personalized subjects (e.g., recognizing a user's pet dog). Human reasoning, in contrast, typically operates within the context of specific subjects in our surroundings. For example, one might ask, "What should I buy for my dog's birthday?"; as opposed to a generic inquiry about "What should I buy for a dog's birthday?". Similarly, when looking at a friend's image, the interest lies in seeing their activities (e.g., "my friend is holding a cat"), rather than merely observing generic human actions (e.g., "a man is holding a cat"). In this paper, we introduce the novel task of personalizing LMMs, so that they can have conversations about a specific subject. We propose Yo'LLaVA, which learns to embed a personalized subject into a set of latent tokens given a handful of example images of the subject. Our qualitative and quantitative analyses reveal that Yo'LLaVA can learn the concept more efficiently using fewer tokens and more effectively encode the visual attributes compared to strong prompting baselines (e.g., LLaVA).

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink

    cs.LG 2025-01 conditional novelty 7.0 of 10

    Adversarial images optimized to induce attention sink behavior increase hallucination rates in multiple MLLMs, including commercial APIs, without visibly degrading response quality.

  2. Improving Personalized Search with Regularized Low-Rank Parameter Updates

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Regularized rank-one LoRA updates to the final value transform of CLIP's text encoder beat textual inversion for personalized retrieval while preserving general knowledge.

  3. POET: Supporting Prompting Creativity and Personalization with Automated Expansion of Text-to-Image Generation

    cs.CV 2025-04 conditional novelty 6.0 of 10

    POET is a tool that finds hidden sameness in AI-generated images, widens those dimensions for more variety, and personalizes future outputs from user feedback.

  4. Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgetting

    cs.CL 2025-04 conditional novelty 5.0 of 10

    Structured Dialogue Fine-Tuning (SDFT) injects domain knowledge into vision-language models through caption, contrastive, and specialization dialogue turns, reporting improved specialization with modest general-capabi...

  5. Personalized Multimodal Large Language Models: A Survey

    cs.CV 2024-12 conditional novelty 4.0 of 10

    The paper provides a survey and taxonomy of personalization techniques for multimodal LLMs across text generation, image generation, recommendation, and retrieval.

Pith tools