Pith. sign in

REVIEW 4 major objections 6 minor 57 references

A Large Vision-Language Model based Environment Perception System for Visually Impaired People

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Injecting segmentation-derived object lists into a vision-language model's prompt reduces hallucinated scene descriptions and improves object existence and count answers, and the authors build this into a wearable assistive system.

desk verdict A useful assistive-system integration whose headline hallucination-reduction claim rests on a benchmark design that lets the model answer from the injected object list rather than from the image. read the letter →

arxiv 2504.18027 v1 pith:UFBSBOIG submitted 2025-04-25 cs.CV cs.AIcs.RO

Zezhou Chen , Zhaoxiang Liu , Kai Wang , Kohou Wang , Shiguo Lian This is my paper · ORCID
classification cs.CVcs.AIcs.RO
keywords largevision-languagemodelhallucinationmitigationsemanticsegmentationassistivetechnologyvisuallyimpaireduserswearabledeviceegocentricsceneunderstandingpromptaugmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that a vision-language model describes a scene more accurately when its prompt includes the object list produced by a semantic segmentation model, because the segmentation reliably states what exists and how many, removing the model's freedom to invent objects. The claim is embedded in a wearable terminal-cloud system for visually impaired people: a long press returns a global spoken description, a tap or swipe announces the object category under the finger with volume indicating distance, and a double tap gives a detailed description of the tapped object. The paper reports that the segmentation-augmented prompt outperforms the baseline LVLM across standard object-hallucination, existence-and-count, and open-ended description benchmarks, and that an exploratory study with twelve visually impaired participants in four everyday scenes rated the system useful and easy to use. A sympathetic reading takes the core contribution to be a training-free, low-cost way to ground language-model scene descriptions in explicit visual evidence.

What carries the argument

The central mechanism is segmentation-grounded prompt augmentation: derive a textual inventory of the scene from the semantic segmentation map and insert it directly into the language model's input prompt before decoding. The inventory sentence is built by traversing the segmented regions, then concatenated with a default query and the aligned visual features extracted by a Vision Transformer (ViT) encoder; the LLM generates the global description conditioned on all three inputs. The same segmentation map powers the interaction layer: a tap or swipe reads the object class at the finger position, the object's pixel area is combined with the depth image to set voice volume, and a double tap crops the object's bounding rectangle to prompt a localized description. The efficiency argument rests on the segmentation model and the LVLM sharing one ViT encoder, so no extra visual feature extraction is needed for the grounding.

What would settle it

Run the augmentation on images where the segmentation model is known to omit or mislabel objects, and check whether the LVLM repeats the wrong list or corrects it; if the model faithfully echoes the segmenter's mistakes, then the reported hallucination reduction is only as strong as the segmentation model and would collapse on out-of-distribution scenes.

Watch

Extended reading notes

Core claim

The core discovery, on the paper's own terms, is that the segmentation result functions as external knowledge for the language model rather than merely as a separate output. The system traverses every segmented object, compiles the categories into one sentence, and places that sentence into the prompt together with the image and the user's query; because the segmentation model specializes in identifying, localizing, and segmenting objects, the prompt now asserts what exists and in what number, and the LVLM is less able to hallucinate absent objects or miscount present ones. The visual encoder is shared between the segmentation decoder and the LVLM, so the grounding costs a reused feature computation instead of a second full-model pass. On benchmark evaluations, the augmented model improves over the baseline in every reported metric: object-hallucination accuracy, precision, recall, and F1; object-existence and count correctness; and the accuracy and detailedness of open-ended scene descriptions.

Load-bearing premise

The method assumes the segmentation model's object list is accurate enough to serve as the ground truth for what is actually in the image, so any object the segmenter misses or mislabels is passed into the prompt as fact and the vision-language model inherits the error.

Editorial extensions

If this is right

  • On the object-hallucination benchmark, the augmentation improves accuracy, precision, recall, and F1 over the baseline in random, popular, and adversarial sampling settings.
  • On the existence-and-count evaluation, the augmented model answers more object-existence questions correctly and sharply improves count correctness, which matters most for assistive scene understanding.
  • On the open-ended description evaluation, the augmented model scores higher on both average accuracy and average detailedness than the baseline.
  • The wearable system provides three gesture-driven retrieval modes with average response times near 4.4 seconds for a global description, 0.4 seconds for a tap or swipe category lookup, and 2.8 seconds for a double-tap object description.
  • The exploratory user study with twelve visually impaired participants in office, shopping-mall, street, and park scenes reports positive usefulness and ease-of-use ratings and supports daily tasks such as obstacle avoidance, way-asking, seat-finding, and photo-taking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond this paper, the same prompt-grounding recipe could be applied to other vision-language models and other dense predictors, such as object detectors or depth maps; testing that would reveal whether the benefit is tied to the chosen segmentation model or generalizes.
  • Because existence and count answers are effectively written into the prompt, the benchmark gains may measure how faithfully the model copies the object list; a stress test with deliberately wrong object lists would show whether the LVLM corrects or inherits the segmenter's errors.
  • The interaction pattern suggests a general assistive-perception principle: use a dense predictor to index the scene spatially, and use a language model only to expand the selected region into words.
  • A concrete extension is to run the same augmentation on robot or drone egocentric captions, where hallucinated objects could cause physical mistakes, to see whether the hallucination reduction transfers outside the assistive setting.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents a wearable assistive system for visually impaired people that combines a semantic segmentation model (SETR) with the LVLM Qwen-VL-Chat. The user captures an image with a phone-connected RGB-D sensor; long-press triggers a global scene description, tap/swipe retrieves object category and distance, and double-tap gives a local object description. The central technical claim is that compiling a sentence listing all segmented objects and injecting it into the LVLM prompt reduces hallucination and improves description quality. The method is evaluated with POPE, MME, and LLaVA-QA90 benchmarks against Qwen-VL-Chat, and with an exploratory user study with twelve visually impaired participants in four scenes.

Significance. The system-level integration is timely and the interaction design is thoughtfully motivated by accessibility constraints (e.g., avoiding earphones, using vibration and volume to convey distance). If the hallucination-reduction mechanism were properly isolated, the training-free prompt augmentation would be a useful low-cost addition. However, the current evidence does not isolate the mechanism: on POPE and MME, the injected object list can directly answer the existence and count questions, so the reported gains may reflect the segmentation model's accuracy rather than a reduction in LVLM hallucination. The LLaVA-QA90 evaluation and the user study also lack the controls and statistical detail needed to support the headline claims. The paper is clearly written and the method is simple enough to reproduce, but the technical experiments need substantial strengthening.

major comments (4)
  1. [Section IV-B, Figure 4, Tables I and II] The POPE and MME evaluations are confounded by construction. The augmented prompt contains a sentence listing every segmented object, while POPE existence questions and MME existence/count questions can be answered by checking that sentence. Therefore the improvements in Table I (e.g., adversarial F1 from 0.830 to 0.851) and Table II (count from 140 to 173) do not establish a reduction in LVLM hallucination, because a lookup oracle on the injected list would produce the same pattern. To support the claim, the authors should include an ablation that removes the object list, an ablation that corrupts it, a condition with ground-truth object annotations as an upper bound, and the standalone object-list accuracy of the SETR model on these benchmarks.
  2. [Section V-B, Table III] The LLaVA-QA90 evaluation is underspecified. The paper reports average accuracy and detailedness but does not describe the scoring rubric, who computed the scores, or inter-rater agreement. Without this information, the claim that the system provides a more accurate description of the scene is not verifiable.
  3. [Section V-C, Figure 7] The exploratory user study has no control condition. All twelve participants evaluated only the proposed system, so the observed positive Likert scores cannot be attributed to the system's specific design; a comparison against, for example, Qwen-VL-Chat without the segmentation prompt or a simpler audio feedback interface is needed. The paper also reports no significance tests or confidence intervals for these scores.
  4. [Tables I-III generally] No measure of variability is reported for any technical benchmark. The POPE improvements are small (e.g., accuracy from 0.842 to 0.862 in the adversarial setting), and without error bars or repeated trials it is unclear whether the differences are reliable. The paper should report bootstrap confidence intervals or significance tests.
minor comments (6)
  1. [Section V-B, POPE paragraph] The text says 'we tested all these 9000 images,' but 9000 refers to question-answer pairs, not images; the preceding sentence correctly reports 1500 images and 9000 question-answer pairs.
  2. [Figure 2 caption] The caption contains typos: 'Chiar' should be 'Chair' and 'he detailed description' should be 'the detailed description'; the capitalization of 'Flowerpot' is also inconsistent.
  3. [Section V-B] The word 'asversarial' is misspelled and should be 'adversarial'; similarly 'testset' should be 'test set'.
  4. [Section IV-B] The claim that the segmentation model shares the same visual encoder with the LVLM needs clarification about whether the SETR encoder uses the same weights as Qwen-VL-Chat's ViT and how feature reuse is implemented at inference time.
  5. [Section II-B and Section V] The related work discusses Woodpecker [54] as a training-free hallucination-correction method, but the experiments do not compare against it, so the claimed efficiency advantage over existing hallucination-mitigation methods is not empirically supported.
  6. [Section V-A] The average response times are reported without context on hardware and network conditions; a brief discussion of variability would help interpret the 'fluent enough' claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found; the POPE/MME prompt-leakage concern is a benchmark-validity issue, not a logical circularity.

full rationale

The paper's central claim is empirical: injecting semantic segmentation results into the LVLM prompt reduces hallucination relative to the Qwen-VL-Chat baseline on POPE, MME, and LLaVA-QA90. The method has no fitted parameters tied to the target claim: the segmentation decoder is fine-tuned on VizWiz, and the LVLM is an off-the-shelf Qwen-VL-Chat model. The 'derivation' is not a mathematical derivation; it is a system design plus benchmark evaluation. The segmentation object list is an input to the prompt, and POPE/MME existence and count questions can be answered directly from that list, so the improvement over the image-only baseline may reflect the segmentation model's object-list accuracy rather than a reduction in LVLM hallucination. This is a benchmark-validity and experimental-isolation concern (Sections IV-B and V-B), not a circularity: the evaluation is not equivalent to the model's definition by construction, no fitted constant is renamed as a prediction, and no load-bearing self-citation or imported uniqueness theorem is used. The paper's self-citations [5] and [12] are prior assistive-device works cited only for context, not as load-bearing evidence. Therefore no circular step is identified.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new particles, forces, dimensions, or conserved quantities. Its central claim rests on the reliability of pretrained vision-language and segmentation models, on the benchmark choice, and on the user-study setting. No scalar constants are fitted to the target benchmarks, aside from the untracked accuracy of the fine-tuned segmentation decoder.

free parameters (1)
  • SETR decoder weights after VizWiz fine-tuning = not reported
    The decoder is fine-tuned on VizWiz to produce the segmentation object list (Section V-A). The reliability of this list is load-bearing for the hallucination-reduction claim, but the paper does not report the resulting segmentation accuracy or training details.
assumptions (5)
  • domain assumption The pretrained Qwen-VL-Chat model can generate useful scene descriptions when given a text prompt.
    Section IV-A selects Qwen-VL-Chat as the backbone; the entire global and local description output rests on this model's capabilities.
  • domain assumption Semantic segmentation results are sufficiently accurate to act as external knowledge in the prompt.
    Section IV-B builds the prompt from the segmentation result, and Section V-B interprets POPE gains as evidence of hallucination reduction, but segmentation errors are never measured.
  • domain assumption Object existence and count questions in POPE and MME capture the hallucination that matters for visually impaired users.
    Section V-B uses these benchmarks to support the system-level claim, yet the system's value also depends on attribute and spatial descriptions.
  • domain assumption Screen coordinates from taps and swipes map correctly to segmentation image coordinates.
    Sections IV-C and IV-D retrieve object category and crop bounding rectangles directly from the tap position.
  • domain assumption Depth values in the RGB-D sensor give reliable relative distances to adjust voice volume.
    Section IV-C averages depth over the segmented object area to set playback volume.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Large Vision-Language Model based Environment Perception System for Visually Impaired People." pith.science (2026). https://pith.science/paper/UFBSBOIG

@misc{pith2026250418027,
  author       = {Pith},
  title        = {Pith review of: A Large Vision-Language Model based Environment Perception System for Visually Impaired People},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UFBSBOIG}},
  note         = {Machine review of arXiv:2504.18027}
}
read the original abstract

It is a challenging task for visually impaired people to perceive their surrounding environment due to the complexity of the natural scenes. Their personal and social activities are thus highly limited. This paper introduces a Large Vision-Language Model(LVLM) based environment perception system which helps them to better understand the surrounding environment, by capturing the current scene they face with a wearable device, and then letting them retrieve the analysis results through the device. The visually impaired people could acquire a global description of the scene by long pressing the screen to activate the LVLM output, retrieve the categories of the objects in the scene resulting from a segmentation model by tapping or swiping the screen, and get a detailed description of the objects they are interested in by double-tapping the screen. To help visually impaired people more accurately perceive the world, this paper proposes incorporating the segmentation result of the RGB image as external knowledge into the input of LVLM to reduce the LVLM's hallucination. Technical experiments on POPE, MME and LLaVA-QA90 show that the system could provide a more accurate description of the scene compared to Qwen-VL-Chat, exploratory experiments show that the system helps visually impaired people to perceive the surrounding environment effectively.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 40 canonical work pages

  1. [10]

    Incorporating external knowledge into machine reading for generative question answering,

    B. Bi, C. Wu, M. Yan, W. Wang, J. Xia, and C. Li, “Incorporating external knowledge into machine reading for generative question answering,” arXiv preprint arXiv:1909.02745 , 2019

    arXiv 1909

  2. [11]

    Check your facts and try again: Improving large language models with external knowledge and auto- mated feedback,

    B. Peng, M. Galley, P. He, H. Cheng, Y . Xie, Y . Hu, Q. Huang, L. Liden, Z. Yu, W. Chen, et al. , “Check your facts and try again: Improving large language models with external knowledge and auto- mated feedback,” arXiv preprint arXiv:2302.12813 , 2023

    arXiv 2023

  3. [54]

    Woodpecker: Hallucination cor- rection for multimodal large language models,

    S. Yin, C. Fu, S. Zhao, et al. , “Woodpecker: Hallucination cor- rection for multimodal large language models,” arXiv preprint arXiv:2310.16045, 2023

    arXiv 2023

  4. [1]

    Ingold, The perception of the environment: Essays in livelihood, dwelling and skill

    T. Ingold, The perception of the environment: Essays in livelihood, dwelling and skill . Psychology Press, 2000

    work page 2000

  5. [2]

    Enabling independent navigation for visually impaired people through a wearable vision-based feedback system,

    H.-C. Wang, R. K. Katzschmann, S. Teng, B. Araki, L. Giarr ´e, and D. Rus, “Enabling independent navigation for visually impaired people through a wearable vision-based feedback system,” in 2017 IEEE international conference on robotics and automation (ICRA) . IEEE, 2017, pp. 6533–6540

    work page 2017

  6. [3]

    Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis,

    R. Bourne et al. , “Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis,” Lancet Global Health, vol. 5, no. 9, pp. 888–897, 2017

    work page 2017

  7. [4]

    Objectively measured visual impairment and dementia prevalence in older adults in the us,

    O. J. Killeen, Y . Zhou, and J. R. Ehrlich, “Objectively measured visual impairment and dementia prevalence in older adults in the us,” JAMA ophthalmology, vol. 141, no. 8, pp. 786–790, 2023

    work page 2023

  8. [5]

    Virtual-blind-road following-based wearable navigation device for blind people,

    J. Bai, S. Lian, Z. Liu, K. Wang, and D. Liu, “Virtual-blind-road following-based wearable navigation device for blind people,” IEEE Trans. Consumer Electron., vol. 64, no. 1, pp. 136–143, 2018

    work page 2018

Show all 57 references
  1. [6]

    Embedded reading device for blind people: A user-centered design,

    J.-P. Peters, C. Thillou, and S. Ferreira, “Embedded reading device for blind people: A user-centered design,” in Appl. Imagery Pattern Recogn. Workshop, 2004, pp. 217–222

    2004

  2. [7]

    Hand-priming in object localization for assistive egocentric vision,

    K. Lee, A. Shrivastava, and H. Kacorri, “Hand-priming in object localization for assistive egocentric vision,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2020, pp. 3422–3432

    2020

  3. [8]

    Vizwiz-fewshot: Locating objects in images taken by people with visual impairments,

    Y .-Y . Tseng, A. Bell, and D. Gurari, “Vizwiz-fewshot: Locating objects in images taken by people with visual impairments,” in European Conference on Computer Vision . Springer, 2022, pp. 575–591

    2022

  4. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly,et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020

    2010 arXiv

  5. [12]

    Smart guiding glasses for visually impaired people in indoor environment,

    J. Bai, S. Lian, Z. Liu, K. Wang, and D. Liu, “Smart guiding glasses for visually impaired people in indoor environment,” IEEE Trans. Consumer Electron., vol. 63, no. 3, pp. 258–266, 2017

    2017

  6. [13]

    An enhanced obstacle avoidance method for the visually impaired using deformable grid,

    M. Kang, S. Chae, J. Sun, S. Lee, and S. Ko, “An enhanced obstacle avoidance method for the visually impaired using deformable grid,” IEEE Trans. Consumer Electron. , vol. 63, no. 2, pp. 169–177, 2017

    2017

  7. [14]

    Collision detection method using image segmentation for the visually impaired,

    S. Chae, M. Kang, J. Sun, B. Kim, and S. Ko, “Collision detection method using image segmentation for the visually impaired,” IEEE Trans. Consumer Electron., vol. 63, no. 4, pp. 392–400, 2017

    2017

  8. [15]

    Tdraw: A computer-based tactile drawing tool for blind people,

    M. Kurze, “Tdraw: A computer-based tactile drawing tool for blind people,” in 2nd Annu. ACM Conf. on Assistive Technol. , 1996, pp. 131–138

    1996

  9. [16]

    Usable gestures for blind people: Understanding preference and performance,

    S. Kane, J. Wobbrock, and R. Ladner, “Usable gestures for blind people: Understanding preference and performance,” in SIGCHI Conf. on Human Factors in Comput. Syst. , 2011, pp. 413–422

    2011

  10. [17]

    How teens with visual impairments take, edit, and share photos on social media,

    C. Bennett et al., “How teens with visual impairments take, edit, and share photos on social media,” in CHI Conf. on Human Factors in Comput. Syst., 2018, p. 76

    2018

  11. [18]

    Flight: A low-cost reading and writing system for economically less-privileged visually-impaired people exploiting ink-based braille system,

    T. Chakraborty, T. Khan, and I. Al, “Flight: A low-cost reading and writing system for economically less-privileged visually-impaired people exploiting ink-based braille system,” in CHI Conf. on Human Factors in Comput. Syst. , 2017, pp. 531–540

    2017

  12. [19]

    Linespace: A sensemaking platform for the blind,

    S. Swaminathan et al. , “Linespace: A sensemaking platform for the blind,” in CHI Conf. on Human Factors in Comput. Syst. , 2016, pp. 2175–2185

    2016

  13. [20]

    Tangible reels: Construction and exploration of tangible maps by visually impaired users,

    J. Ducasse, M. Serrano, and C. Jouffrais, “Tangible reels: Construction and exploration of tangible maps by visually impaired users,” in CHI Conf. on Human Factors in Comput. Syst. , 2016, pp. 2186–2197

    2016

  14. [21]

    Comparing computer- based drawing methods for blind people with real-time tactile feed- back,

    J. Bornschein, D. Bornschein, and G. Weber, “Comparing computer- based drawing methods for blind people with real-time tactile feed- back,” in CHI Conf. on Human Factors in Comput. Syst. , 2018, p. 115

    2018

  15. [22]

    Accessible maps for the blind: comparing 3d printed models with tactile graphics,

    L. Holloway, K. Marriott, and M. Butler, “Accessible maps for the blind: comparing 3d printed models with tactile graphics,” in CHI Conf. on Human Factors in Comput. Syst. , 2018, p. 198

    2018

  16. [23]

    Taking into account sensory knowledge: The case of geo-techologies for children with visual impairments,

    E. Brul ´e and G. Bailly, “Taking into account sensory knowledge: The case of geo-techologies for children with visual impairments,” in CHI Conf. on Human Factors in Comput. Syst. , 2018, p. 236

    2018

  17. [24]

    People with visual impairment training personal object recognizers: feasibility and challenges,

    H. Kacorri, K. Kitani, J. Bigham, and C. Asakawa, “People with visual impairment training personal object recognizers: feasibility and challenges,” in CHI Conf. on Human Factors in Comput. Syst. , 2017, pp. 5839–5849

    2017

  18. [25]

    Synthesizing stroke gestures across user populations: A case for users with visual im- pairments,

    L. Leiva, D. Mart ´ın-Albo, and R.-D. Vatavu, “Synthesizing stroke gestures across user populations: A case for users with visual im- pairments,” in CHI Conf. on Human Factors in Comput. Syst. , 2017, pp. 4182–4193

    2017

  19. [26]

    A face recognition application for people with visual impairments: Understanding use beyond the lab,

    Y . Zhao, S. Wu, L. Reynolds, and S. Azenkot, “A face recognition application for people with visual impairments: Understanding use beyond the lab,” in CHI Conf. on Human Factors in Comput. Syst. , 2018, p. 215

    2018

  20. [27]

    A multitask grocery assist system for the visually impaired: Smart glasses, gloves, and shopping carts provide auditory and tactile feedback,

    S. Advani et al. , “A multitask grocery assist system for the visually impaired: Smart glasses, gloves, and shopping carts provide auditory and tactile feedback,” IEEE Trans. Consum. Electron. , vol. 6, no. 1, pp. 73–81, 2017

    2017

  21. [28]

    Hindsight: Enhancing spa- tial awareness by sonifying detected objects in real-time 360-degree video,

    E. Schoop, J. Smith, and B. Hartmann, “Hindsight: Enhancing spa- tial awareness by sonifying detected objects in real-time 360-degree video,” in CHI Conf. on Human Factors in Comput. Syst. , 2018, p. 143

    2018

  22. [29]

    You only look once: Unified, real-time object detection,

    J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in IEEE Conf. Comput. Vision Pattern Recogn. (CVPR) , 2016, pp. 779–788

    2016

  23. [30]

    Ssd: Single shot multibox detector,

    W. Liu et al. , “Ssd: Single shot multibox detector,” in Eur. Conf. on Comput. Vision (ECCV) , 2016, pp. 21–37

    2016

  24. [31]

    Rich feature hierarchies for accurate object detection and semantic segmentation,

    R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in IEEE Conf. Comput. Vision Pattern Recogn. (CVPR) , 2014, pp. 580–587

    2014

  25. [32]

    Faster r-cnn: Towards real- time object detection with region proposal networks,

    S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real- time object detection with region proposal networks,” IEEE Trans. Pattern Anal. Machine Intell. , vol. 39, no. 6, pp. 1137–1149, 2017

    2017

  26. [33]

    Fully convolutional networks for semantic segmentation,

    J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in IEEE Conf. Comput. Vision Pattern Recogn. (CVPR), 2015, pp. 3431–3440

    2015

  27. [34]

    Segnet: A deep convolutional encoder-decoder architecture for scene segmentation,

    V . Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for scene segmentation,” IEEE Trans. Pattern Anal. Machine Intell. , vol. PP, no. 99, pp. 1–1, 2017

    2017

  28. [35]

    Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,

    L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE Trans. Pattern Anal. Machine Intell. , vol. 40, no. 4, pp. 834–848, 2018

    2018

  29. [36]

    Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,

    S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y . Wang, Y . Fu, J. Feng, T. Xiang, P. H. Torr, et al., “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- ti...

    2021

  30. [37]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 10 012–10 022

    2021

  31. [38]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  32. [39]

    Fusenet: In- corporating depth into semantic segmentation via fusion-based cnn architecture,

    C. Hazirbas, L. Ma, C. Domokos, and D. Cremers, “Fusenet: In- corporating depth into semantic segmentation via fusion-based cnn architecture,” in Asian Conf. on Comput. Vision (ACCV) , 2016

    2016

  33. [40]

    Rednet: Residual encoder- decoder network for indoor rgb-d semantic segmentation,

    J. Jiang, L. Zheng, F. Luo, and Z. Zhang, “Rednet: Residual encoder- decoder network for indoor rgb-d semantic segmentation,” arXiv preprint, p. arXiv:1806.01054, 2018

    2018 arXiv

  34. [41]

    Show and tell: A neural image caption generator,

    O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in IEEE Conf. Comput. Vision Pattern Recogn. (CVPR), 2015, pp. 3156–3164

    2015

  35. [42]

    Knowing when to look: Adaptive attention via a visual sentinel for image captioning,

    J. Lu, C. Xiong, D. Parikh, and R. Socher, “Knowing when to look: Adaptive attention via a visual sentinel for image captioning,” in IEEE Conf. Comput. Vision Pattern Recogn. (CVPR) , 2017, pp. 3242–3250

    2017

  36. [43]

    Show, attend and tell: Neural image caption generation with visual attention,

    K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y . Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in International conference on machine learning . PMLR, 2015, pp. 2048–2057

    2015

  37. [44]

    Dense captioning with joint inference and visual context,

    L. Yang, K. Tang, J. Yang, and L.-J. Li, “Dense captioning with joint inference and visual context,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 2193–2202

    2017

  38. [45]

    Unifying vision-and-language tasks via text generation,

    J. Cho, J. Lei, H. Tan, and M. Bansal, “Unifying vision-and-language tasks via text generation,” in International Conference on Machine Learning. PMLR, 2021, pp. 1931–1942

    2021

  39. [46]

    Multimodal few-shot learning with frozen language models,

    M. Tsimpoukelli, J. L. Menick, S. Cabi, S. Eslami, O. Vinyals, and F. Hill, “Multimodal few-shot learning with frozen language models,” Advances in Neural Information Processing Systems, vol. 34, pp. 200– 212, 2021

    2021

  40. [47]

    Simvlm: Simple visual language model pretraining with weak supervision,

    Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y . Tsvetkov, and Y . Cao, “Simvlm: Simple visual language model pretraining with weak supervision,” arXiv preprint arXiv:2108.10904 , 2021

    2021 arXiv

  41. [48]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems , vol. 36, 2024

    2024

  42. [49]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” arXiv preprint arXiv:2301.12597 , 2023

    2023 arXiv

  43. [50]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. , “Flamingo: a visual language model for few-shot learning,” Advances in Neural Information Processing Systems , vol. 35, pp. 23 716–23 736, 2022

    2022

  44. [51]

    Detecting and preventing hallucinations in large vision language models,

    A. Gunjal, J. Yin, and E. Bas, “Detecting and preventing hallucinations in large vision language models,” arXiv preprint arXiv:2308.06394 , 2023

    2023 arXiv

  45. [52]

    Mitigating hallucination in large multi-modal models via robust instruction tuning,

    F. Liu, K. Lin, L. Li, et al. , “Mitigating hallucination in large multi-modal models via robust instruction tuning,” arXiv preprint arXiv:2306.14565, 2023

    2023 arXiv

  46. [53]

    Evaluation and mitigation of agnosia in multimodal large language models,

    J. Lu, J. Rao, K. Chen, et al., “Evaluation and mitigation of agnosia in multimodal large language models,” arXiv preprint arXiv:2309.04041, 2023

    2023 arXiv

  47. [55]

    Qwen-vl: A versatile vision-language model for under- standing, localization, text reading, and beyond,

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A versatile vision-language model for under- standing, localization, text reading, and beyond,” 2023

    2023

  48. [56]

    Evaluating object hallucination in large vision-language models,

    Y . Li, Y . Du, K. Zhou, et al., “Evaluating object hallucination in large vision-language models,” arXiv preprint arXiv:2305.10355 , 2023

    2023 arXiv

  49. [57]

    Mme: A comprehensive evaluation benchmark for multimodal large language models,

    C. Fu, P. Chen, Y . Shen, et al. , “Mme: A comprehensive evaluation benchmark for multimodal large language models,” arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.