VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairments

Bufang Yang; Kaiwei Liu; Lixing He; Zhenyu Yan

VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairments

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.02508 v1 pith:MDIIHZKZ submitted 2024-04-03 cs.CV cs.AIcs.LG

Bufang Yang , Lixing He , Kaiwei Liu , Zhenyu Yan This is my paper · ORCID

classification cs.CVcs.AIcs.LG

keywords visualindividualsmllmsviassistimagesimpairmentsanswerscapabilities

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

Individuals with visual impairments, encompassing both partial and total difficulties in visual perception, are referred to as visually impaired (VI) people. An estimated 2.2 billion individuals worldwide are affected by visual impairments. Recent advancements in multi-modal large language models (MLLMs) have showcased their extraordinary capabilities across various domains. It is desirable to help VI individuals with MLLMs' great capabilities of visual understanding and reasoning. However, it is challenging for VI people to use MLLMs due to the difficulties in capturing the desirable images to fulfill their daily requests. For example, the target object is not fully or partially placed in the image. This paper explores how to leverage MLLMs for VI individuals to provide visual-question answers. VIAssist can identify undesired images and provide detailed actions. Finally, VIAssist can provide reliable answers to users' queries based on the images. Our results show that VIAssist provides +0.21 and +0.31 higher BERTScore and ROUGE scores than the baseline, respectively.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

EmoAssist: Emotional Assistant for Visual Impairment Community
cs.CV 2025-02 conditional novelty 6.0 of 10

A fine-tuned LLaVA model, trained with preference optimization on 800 emotional image-QA examples, beats GPT-4o on a new empathy-focused benchmark for visual impairment assistance.

Pith tools