Pith. sign in

REVIEW 5 major objections 5 minor 49 references

The paper introduces VIABench, a 47-hour first-person video benchmark built largely from footage recorded by blind individuals, and claims that today's strongest multimodal models fall far short of usable blind-assistance systems: the best

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:26 UTC pith:LQREVIT5

load-bearing objection VIABench is a genuine step forward in assistive-vision benchmarking, but the temporal ground truth hinges on an unvalidated 'safe alert range' that the paper never stress-tests. the 5 major comments →

arxiv 2607.14660 v1 pith:LQREVIT5 submitted 2026-07-16 cs.CV

VIABench: A Comprehensive Video Benchmark Collected from Blind Individuals for Visual Impairment Assistance

Yunfeng Liu , Yuandong Yang , Jiarui Han , Zhenpeng Huang , Yuqing Tang , Xiangyu Zeng , Gangshan Wu , Limin Wang This is my paper
classification cs.CV
keywords visual impairment assistanceegocentric videomultimodal large language modelsproactive remindervideo benchmarktemporal localizationTPAD
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces VIABench, a 47-hour egocentric video benchmark drawn mostly from videos recorded by blind individuals, with 14,526 annotations spanning three assistance tasks: proactively warning about navigation hazards, answering visual questions, and giving step-by-step interaction guidance. Its central claim is that current multimodal language models are far from usable as blind-assistance systems: the strongest evaluated model reaches an overall score of 28.8 out of 100, and offline models recall fewer than 45 percent of reminder-triggering moments. To make offline models testable at all, the paper contributes TPAD, a lightweight decoding technique that converts a single forward pass into frame-level alert probabilities, running about 4.5 times faster than conventional frame-by-frame prompting. The paper also shows that online streaming models achieve high recall only by narrating continuously rather than by understanding when a warning is genuinely required, and that realistic assistance evaluation needs to distinguish 'when to act' from 'how to answer.'

Core claim

On its own terms, the paper demonstrates that state-of-the-art multimodal large language models are not yet capable of real-world visual assistance for blind users. Across all three tasks, even the strongest model averages 28.8 out of 100; in the central Proactive Reminder task, offline models retrieve fewer than 45 percent of the annotated warning intervals, and online streaming models that do retrieve many intervals do so by producing dense, unsolicited narration rather than task-aligned alerts. A controlled stage-2 ablation, where all models see the same 32-frame window, shows that generation quality also degrades sharply on direction-tracking tasks, and qualitative examples reveal a recu

What carries the argument

The load-bearing mechanism is TPAD (Token-Level Prompt Activation Decoding), which takes a fixed forced-choice prompt such as 'Should the user be warned? (A) Yes; (B) No,' feeds the prompt and all video frames as one concatenated token sequence, then reads the hidden state of each frame's final token and projects it through the language-modeling head to obtain a per-frame alert probability. This turns an offline MLLM into a proactive detector without fine-tuning and with full preceding-video context in a single forward pass. The accompanying evaluation machinery includes the Proactive Detection Rate (PDR), which counts an alert as correct only if it falls inside the annotated ground-truth in

Load-bearing premise

The timing labels for when a warning should fire come from a fixed hand-defined 'safe alert range' (1.5–2 meters ahead, 50 centimeters to each side) that is asserted without citation or sensitivity analysis; if that assumption about safe human walking distance is wrong, every Proactive Detection Rate and the headline conclusion shift.

What would settle it

Re-score a random sample of VIABench videos while varying the safe alert range across plausible values (e.g., front 1–3 m, lateral 30–70 cm) and check whether recall rankings and the sub-45% offline ceiling survive; a stronger test would be to have blind users wear a prototype and mark the moments they actually want an alert, then compare those annotations to the benchmark's intervals—if the distributions diverge substantially, the benchmark's ground truth does not represent real warning needs.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • TPAD makes offline multimodal models evaluable for proactive, real-time tasks without any training, and its 4.5x speedup suggests a practical path for benchmarking larger models on long, continuous video.
  • Because adding more input frames yields only marginal gains on VIABench, the paper implies that assistive video understanding is dominated by immediate, situational perception rather than long-horizon memory—an argument that small, low-latency models may be the right deployment target.
  • The sharp gap between online models' high retrieval recall and their poor end-to-end scores implies that output frequency alone is not evidence of proactive competence; evaluation metrics need to penalize irrelevant continuous narration.
  • The consistently low offline recall across all sub-tasks sets an upper bound on end-to-end assistive quality, pointing to alert-timing retrieval as the primary bottleneck for future model development rather than language generation alone.
  • The three-task structure (reminder, VQA, interaction) provides a template for evaluating assistive systems along the axes of proactive perception, reactive reasoning, and interactive guidance, which generic video-QA benchmarks do not cover.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The hand-defined 'safe alert range' (1.5–2 m ahead, 50 cm lateral) is asserted without citation or sensitivity analysis; if safe warning distance varies by walking speed, height, or cane technique, then every PDR score and cross-model comparison could shift. A per-user or per-speed calibration of alert intervals would make the benchmark more robust and its conclusions more transportable.
  • The finding that frame count barely helps raises a testable extension: a deliberately 'memory-free' model that reasons only on the current second of video may match or exceed models that consume long context, at a fraction of the compute cost—a hypothesis the paper's Table 5 already hints at but does not directly pursue.
  • The observation that open-source models give sighted-user-style instructions suggests a concrete training signal: explicitly conditioning models on the user's blindness (e.g., read-out expectations, camera-adjustment guidance) could be evaluated as a separate capability axis, and likely explains part of the interaction-task gap.
  • Because online models can inflate recall by narrating constantly, future benchmarks may need precision- or cost-aware variants, where a false-alert penalty reflects the real-world cost of distracting a blind user during navigation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces VIABench, a video benchmark for evaluating MLLMs as assistants for visually impaired individuals (VIIs), built from 761 egocentric videos (46.9 hours, 14,526 annotations) predominantly recorded by blind users. It defines three tasks: Proactive Reminder (21 sub-tasks), VQA, and Vision-Guided Interaction. The authors propose TPAD, a token-level decoding mechanism that converts offline MLLMs into frame-wise proactive alert detectors. Experiments on multiple open/closed/online models show GPT-5 scores only 28.8 overall, with offline stage-1 recall below 45%, and conclude current MLLMs are inadequate for proactive blind assistance.

Significance. If the benchmark's construct validity holds, VIABench fills a genuine gap: prior datasets (VizWiz, EgoBlind, WalkVLM) are image-based, QA-only, or very short clips. The long-form first-person video, time-aligned multi-task annotations, and explicit robustness dimension are valuable resources. The paper ships a substantial dataset and an efficient TPAD evaluation method, and the finding that online models' high recall is spurious (dense captioning) is a useful caution. However, the quantitative claims are currently under-supported: no inter-annotator reliability, no sensitivity analysis for the alert-timing rule, and an incompletely specified aggregation metric. The central conclusion may survive, but the evidence needs strengthening.

major comments (5)
  1. [Appendix 9.2 and Eq. (3)] Appendix 9.2 defines the 'safe alert range' (front 1.5–2 m, lateral 50 cm) with no citation or sensitivity analysis. This range determines the ground-truth intervals [t_s, t_e] used in Eq. (3) for PDR and therefore all Proactive Reminder scores in Tables 2–4. The central conclusion that models 'struggle' depends on this timing standard. If, for example, the front range is actually 3 m or the lateral range 1 m, early warnings currently counted as false would become true, altering rank order and absolute scores. Please provide a sensitivity analysis over plausible ranges, or cite and validate against orientation-and-mobility guidelines.
  2. [Section 5.2 and Appendix 7.2–7.4] The headline 'Avg' and 'Overall' scores lack a precise definition. The text says the Proactive Reminder score is a 'weighted combination of recall and response correctness' but no weights or aggregation formula are given. PDR (Eq. 3) and MPS (Eq. 4) are separate; Table 2 reports a number per sub-task and an Avg, but the mapping is unreported. Without an explicit metric definition, the results in Table 2 and the 28.8 overall figure are not reproducible.
  3. [Section 10] Section 10 describes annotation training and QA but reports no inter-annotator agreement (e.g., Cohen's kappa or temporal IoU) on start/end timestamps, task labels, or descriptions. Given the benchmark's core value is fine-grained temporal annotation, reliability statistics on a subset are essential to establish that intervals are not idiosyncratic. Without them, the ground-truth intervals are unvalidated.
  4. [Sections 5.2, 7.3, 7.4] GPT-5 is used to judge outputs, including GPT-5's own outputs. Although task-specific criteria-guided prompts mitigate generic-similarity bias, no validation of the judge against human ratings is reported. Please report a human-validated subset (e.g., 100 samples) with correlation/agreement, and ideally use a different model as judge when scoring GPT-5.
  5. [Section 4.2 and Appendix 8.5] TPAD is the mechanism by which all offline models produce Proactive Reminder scores, yet its validity rests on the untested premise that token-level hidden states of a forced-choice prompt reflect frame-level alert-worthiness. The only evidence is one 53-second clip (Section 8.5) where TPAD matches conventional prompting; no comparison against ground-truth alert intervals or against online models' labels is provided. A systematic validation (e.g., on a random subset of VIABench) is needed before the offline-model PDR numbers can be interpreted.
minor comments (5)
  1. [Throughout] Model names are inconsistent: 'LLaV A-OneVision-1.5' and 'LLaV A-Video-7B' should be 'LLaVA-OneVision-1.5' and 'LLaVA-Video-7B'; similar spacing issues appear in Table 2 and the text.
  2. [Table 1 caption] Column abbreviations 'Anno.', 'TA', and 'Robust' are not expanded in the caption; please define them for readability.
  3. [References] Reference [19] is a blog post URL; if a technical report or documentation page exists, cite that instead.
  4. [Appendix 9.2] 'task list 6' should be 'Table 6'.
  5. [Figure 10] The red bounding box in the left panel is not clearly visible in the printed figure; consider adding explicit dimension labels.

Circularity Check

0 steps flagged

No significant circularity; the benchmark's ground-truth definitions and evaluation metrics are external to the models being tested, and the cited concerns are construct-validity limitations rather than derivation loops.

full rationale

VIABench is a dataset-and-benchmark paper rather than a predictive derivation, and I find no step where a claimed result reduces by construction to its own inputs. The Proactive Reminder ground truth is produced by human annotators using explicit task definitions (Sec. 10.2) and a hand-specified 'safe alert range' (Sec. 9.2); while that range is an uncited and unsensitivity-analyzed annotation rule, it is not a fitted parameter and does not encode the paper's conclusion that MLLMs underperform. The PDR and MPS metrics (Eqs. 3-4) are standard recall/similarity measures computed against these human labels, and the VQA/VGI scores (Eq. 5) compare model outputs to human ground-truth text; even though GPT-5 serves both as judge and as an evaluated model, its scores are not forced to any value by definition (GPT-5 scores 62.9 on VQA, not 1.0), so this is an evaluator-bias limitation rather than a circular reduction. TPAD is a proposed inference-time adaptation mechanism with no parameters fitted to the benchmark's ground-truth intervals. The same-author citations to CG-Bench and MVBench appear only in related-work comparisons and are not load-bearing for any claim. The paper's own Limitations section acknowledges the VGI simulation domain gap, which is an external-validity caveat, not a circularity. Overall, the central benchmark construction and evaluation are self-contained: the models are external, the labels are human-authored, and the conclusions follow from the measured scores rather than from the annotation definitions themselves.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central quantitative claims rest on hand-set alert-range thresholds, an undisclosed score-weighting scheme, and a self-referential GPT-5 judge, plus the standard TPAD softmax assumption. There are no newly invented physical entities; the 'safe alert range' is a measurement rule rather than an object.

free parameters (5)
  • safe alert range thresholds = front 1.5–2 m; lateral 50 cm
    Defines the ground-truth alert window for all Proactive Reminder annotations (Appendix 9.2). Hand-chosen anthropometric values with no cited source or sensitivity analysis; all PDR scores depend on this window.
  • Proactive Reminder score combination weights = not specified
    Main results Table 2 report 'Avg' as a 'weighted combination of recall and response correctness' (Section 5.2), but no weights or formula appear in the paper or appendix; headline numbers are under-specified.
  • Direction Deviation preset threshold = not specified
    Task definition DD requires alert when 'lateral drift or angular deviation exceeds a preset threshold' (Appendix Table 6), but the threshold value is never given; the task is not reproducible from the definition.
  • TPAD frame sampling rate = 1 FPS
    Appendix 7.1 fixes 1 FPS for TPAD and GPT evaluations. Alert timing and PDR are sensitive to this rate; no ablation on sampling rate is provided for the main results.
  • task-specific distance thresholds (SLUD: within 3 m; SD: 30–50 cm; OA: 1.5–2 m) = various
    Task definitions embed hand-picked distances that determine what counts as a correct alert; these are annotation design choices not derived from measurement or literature.
axioms (5)
  • ad hoc to paper Token-level hidden states from a language modeling head projection reflect frame-level alert-worthiness for a forced-choice prompt.
    TPAD (Section 4.2, Eq. 1–2) asserts that softmax over 'A'/'B' logits for each frame's final token yields a reliable alert probability. No validation against true proactive behavior beyond one clip in the appendix.
  • domain assumption GPT-5 produces consistent, valid similarity/harmfulness scores for VQA, VGI, and reminder descriptions.
    Used as judge in Eqs. (4)–(5) and Table 2; no human-annotation agreement, no calibration, and no ablations over judge prompts; GPT-5 is itself one of the evaluated models.
  • domain assumption Vendor annotations are accurate and consistent without reported inter-annotator agreement metrics.
    Section 3.2 describes pilot + vendor annotation with QA loops, but no IAA statistics; the temporal GT intervals are the sole basis of PDR.
  • domain assumption Videos from blind creators on YouTube/Bilibili/DouYin are representative of blind-assistance tasks and ethically usable for public release.
    Section 3.2 Step 1: sourced via keyword search and manual screening; no IRB/consent statement for re-distribution of footage of a vulnerable population.
  • standard math Standard softmax and language-model decoding mathematics are valid as used in Eq. (1)–(2).
    Background for TPAD's probability computation.

pith-pipeline@v1.3.0-alltime-deepseek · 26058 in / 13977 out tokens · 140255 ms · 2026-08-02T01:26:31.868648+00:00 · methodology

0 comments
read the original abstract

Visually impaired individuals (VIIs) encounter significant daily challenges due to limited access to visual information. Although Multimodal Large Language Models (MLLMs) have achieved impressive results on general vision and language tasks, their practical utility in real-world blind assistance still remains largely underexplored. To fill this gap, we introduce VIABench, a comprehensive video benchmark specifically designed to evaluate MLLMs in Visually Impaired Assistance scenarios using first-person videos recorded or shared by VIIs themselves. VIABench defines three core tasks, each targeting a distinct requirement in visual assistance. Proactive Reminder: Assesses the model's ability to interpret ongoing video content while proactively anticipating and verbally describing upcoming navigation-critical events; Visual Question Answering (VQA): Evaluates the model's capacity to answer user-posed questions about the environment or objects within the video; Vision-Guided Interaction: Tests context-aware reasoning to accomplish intentional interactions between user and environment. To ensure a robust and fair evaluation, we propose a rigorous benchmarking pipeline that supports both online (real-time) and offline settings. Our experiments demonstrate that current MLLMs still struggle to deliver comprehensive support for VIIs, especially in the Proactive Reminder task, which demands accurate anticipation and real-time responsiveness. We hope VIABench will drive future research toward developing customized MLLMs for real-world assistance, ultimately improving navigation and interaction experiences for visually impaired individuals. Code and data will be released at https://github.com/MCG-NJU/VIABench.

Figures

Figures reproduced from arXiv: 2607.14660 by Gangshan Wu, Jiarui Han, Limin Wang, Xiangyu Zeng, Yuandong Yang, Yunfeng Liu, Yuqing Tang, Zhenpeng Huang.

Figure 1
Figure 1. Figure 1: Task definitions in VIABench. Our benchmark contains 3 core tasks for visual impairment assistance, corresponding to three types of user behavior when interacting with the model: (1) doing nothing, (2) asking a question, and (3) receiving and acting on iterative guidance to complete a goal. We highlight the object relevant to the annotation in a red box for better visibility. time-aligned, egocentric video… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the VIABench data generation pipeline. The process consists of five key stages: (1) Sourcing Videos, (2) Defining [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (Up) The word cloud of annotations highlights a strong focus on navigation and obstacle detection. (Down) A histogram illustrating the diverse distribution of video lengths, with a notable presence of long-form content. in real-world assistive needs. It comprises 530 real VII videos (∼44 hours) and 231 informed simulated videos designed to complete the coverage of the Vision-Guided Interaction task. The se… view at source ↗
Figure 4
Figure 4. Figure 4: Overview of Token-Level Prompt Activation Decoding [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Inference Latency (y-axis) v.s. Frames Number (x-axis). [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 8
Figure 8. Figure 8: Runtime comparison on a 53s video segment with 6 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 6
Figure 6. Figure 6: Example of hallucination in open-source multimodal models. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Model responses to the request: “Please help me check the remaining time on the washing machine.” While the ground truth, [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Responses of the LiveCC model on the “Obstacle Alert” [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: (Left) Visualization of the defined safe alert range (red bounding box). (Right) Examples of alert-timing categories: targets requiring immediate alerts (red), targets that should be alerted later when the user approaches them (blue), and targets that do not require alerts (yellow). Special cases: In certain edge cases, alerts are still re￾quired even when targets fall outside the predefined safe zone, pr… view at source ↗
Figure 12
Figure 12. Figure 12: Prompt templates for VQA and vision-grounded interaction (VGI) tasks. [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Prompt templates for scoring three main tasks [PITH_FULL_IMAGE:figures/full_fig_p022_13.png] view at source ↗
Figure 15
Figure 15. Figure 15: Task examples of Proactive Reminder [OA] Obstacle Alert Task Definition: Detect and describe nearby obstacles within 1.5 to 2 meters ahead or 0.5 meters to the sides of the user. Obstacles include people, vehicles, poles, curbs, walls, and other physical objects. Exclude guide dogs and white canes used by the blind. Start time = 0.89s End time = 1.85s Ground Truth: Time Interval: [0.89s, 1.85s] Descriptio… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 20 linked inside Pith

  1. [1]

    Llava-onevision-1.5: Fully open framework for democratized multimodal training.arXiv preprint arXiv:2509.23661, 2025

    Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Chunsheng Wu, et al. Llava-onevision-1.5: Fully open framework for democratized multimodal training.arXiv preprint arXiv:2509.23661, 2025. 7, 8, 2, 3

    Pith/arXiv arXiv 2025

  2. [2]

    Qwen2.5- vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5- vl technical report, ...

    2025

  3. [3]

    Cg-bench: Clue-grounded question answering benchmark for long video understanding

    Guo Chen, Yicheng Liu, Yifei Huang, Baoqi Pei, Jilan Xu, Yuping He, Tong Lu, Yali Wang, and Limin Wang. Cg-bench: Clue-grounded question answering benchmark for long video understanding. InICLR, 2025. 1, 3

    2025

  4. [4]

    Videollm-online: Online video large language model for streaming video

    Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18407–18418, 2024. 6, 7, 8, 3

    2024

  5. [5]

    Livecc: Learning video llm with streaming speech transcription at scale

    Joya Chen, Ziyun Zeng, Yiqi Lin, Wei Li, Zejun Ma, and Mike Zheng Shou. Livecc: Learning video llm with streaming speech transcription at scale. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 29083– 29095, 2025. 7, 8, 2, 3

    2025

  6. [6]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long con- text, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long con- text, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025. 7, 8, 3, 4, 6

    Pith/arXiv arXiv 2025

  7. [7]

    An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020. 6

    Pith/arXiv arXiv 2010

  8. [8]

    Mmbench-video: A long-form multi-shot benchmark for holistic video under- standing.Advances in Neural Information Processing Sys- tems, 37:89098–89124, 2024

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video under- standing.Advances in Neural Information Processing Sys- tems, 37:89098–89124, 2024. 10

    2024

  9. [9]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024. 1, 3

    Pith/arXiv arXiv 2024

  10. [10]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18995–19012, 2022. 8

    2022

  11. [11]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  12. [12]

    Guidedog: A real-world egocentric multimodal dataset for blind and low-vision accessibility-aware guidance.arXiv preprint arXiv:2503.12844, 2025

    Junhyeok Kim, Jaewoo Park, Junhee Park, Sangeyl Lee, Ji- wan Chung, Jisung Kim, Ji Hoon Joung, and Youngjae Yu. Guidedog: A real-world egocentric multimodal dataset for blind and low-vision accessibility-aware guidance.arXiv preprint arXiv:2503.12844, 2025. 3

    Pith/arXiv arXiv 2025

  13. [13]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video understanding benchmark. InCVPR, pages 22195–22206,

  14. [14]

    VideoChat: chat-centric video understanding.Sci

    Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. VideoChat: chat-centric video understanding.Sci. China Inf. Sci., 68(10),

  15. [15]

    Ovo-bench: How far is your video-llms from real-world online video understanding?arXiv preprint arXiv:2501.05510, 2025

    Yifei Li, Junbo Niu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, et al. Ovo-bench: How far is your video-llms from real-world online video understanding?arXiv preprint arXiv:2501.05510, 2025. 3, 6

    Pith/arXiv arXiv 2025

  16. [16]

    Streamingbench: Assessing the gap for mllms to achieve streaming video un- derstanding.arXiv preprint arXiv:2411.03628, 2024

    Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. Streamingbench: Assessing the gap for mllms to achieve streaming video un- derstanding.arXiv preprint arXiv:2411.03628, 2024. 3

    Pith/arXiv arXiv 2024

  17. [17]

    Soccernet-caption: Dense video captioning for soccer broadcasts commentaries

    Hassan Mkhallati, Anthony Cioppa, Silvio Giancola, Bernard Ghanem, and Marc Van Droogenbroeck. Soccernet-caption: Dense video captioning for soccer broadcasts commentaries. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5074–5085, 2023. 8

    2023

  18. [18]

    Hello gpt-4o

    OpenAI. Hello gpt-4o. https://openai.com/index/ hello-gpt-4o/, 2024. 4, 7, 8

    2024

  19. [19]

    Introducing gpt-5

    OpenAI. Introducing gpt-5. https://openai.com/zh- Hans-CN/index/introducing-gpt-5/ , 2025. 7, 8, 3, 4, 6

    2025

  20. [20]

    Video question answering for people with visual impair- 9 ments using an egocentric 360-degree camera.arXiv preprint arXiv:2405.19794, 2024

    Inpyo Song, Minjun Joo, Joonhyung Kwon, and Jangwon Lee. Video question answering for people with visual impair- 9 ments using an egocentric 360-degree camera.arXiv preprint arXiv:2405.19794, 2024. 3

    Pith/arXiv arXiv 2024

  21. [21]

    Gemini 1.5: Unlocking mul- timodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking mul- timodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. 1, 3

    Pith/arXiv arXiv 2024

  22. [22]

    Kimi-vl technical report.arXiv preprint arXiv:2504.07491, 2025

    Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report.arXiv preprint arXiv:2504.07491, 2025. 7, 8, 2, 3, 4

    Pith/arXiv arXiv 2025

  23. [23]

    Dynamic crosswalk scene understanding for the visually impaired.IEEE transactions on neural systems and rehabilitation engineering, 29:1478–1486, 2021

    Shishun Tian, Minghuo Zheng, Wenbin Zou, Xia Li, and Lu Zhang. Dynamic crosswalk scene understanding for the visually impaired.IEEE transactions on neural systems and rehabilitation engineering, 29:1478–1486, 2021. 1

    2021

  24. [24]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 6

    2017

  25. [25]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 1, 3

    Pith/arXiv arXiv 2024

  26. [26]

    Lvbench: An extreme long video understanding benchmark.arXiv preprint arXiv:2406.08035, 2024

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark.arXiv preprint arXiv:2406.08035, 2024. 1, 3

    Pith/arXiv arXiv 2024

  27. [27]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. 7, 8, 2, 3, 4, 6

    Pith/arXiv arXiv 2025

  28. [28]

    A dataset for the visually impaired walk on the road.Displays, 79:102486, 2023

    Haiying Xia, Cong Yao, Yumei Tan, and Shuxiang Song. A dataset for the visually impaired walk on the road.Displays, 79:102486, 2023. 1

    2023

  29. [29]

    Egoblind: Towards egocentric visual assistance for the blind people

    Junbin Xiao, Nanxin Huang, Hao Qiu, Zhulin Tao, Xun Yang, Richang Hong, Meng Wang, and Angela Yao. Egoblind: Towards egocentric visual assistance for the blind people. arXiv preprint arXiv:2503.08221, 2025. 1, 3, 4

    arXiv 2025

  30. [30]

    Streamingvlm: Real-time understanding for infinite video streams.arXiv preprint arXiv:2510.09608, 2025

    Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Kelly Peng, Yao Lu, and Song Han. Streamingvlm: Real-time understanding for infinite video streams.arXiv preprint arXiv:2510.09608, 2025. 7, 8, 3

    Pith/arXiv arXiv 2025

  31. [31]

    Vias- sist: Adapting multi-modal large language models for users with visual impairments

    Bufang Yang, Lixing He, Kaiwei Liu, and Zhenyu Yan. Vias- sist: Adapting multi-modal large language models for users with visual impairments. In2024 IEEE International Work- shop on Foundation Models for Cyber-Physical Systems & Internet of Things (FMSys), pages 32–37. IEEE, 2024. 3

    2024

  32. [32]

    Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024. 1, 3, 7, 8

    Pith/arXiv arXiv 2024

  33. [33]

    Lytnet: A convolu- tional neural network for real-time pedestrian traffic lights and zebra crossing recognition for the visually impaired

    Samuel Yu, Heon Lee, and John Kim. Lytnet: A convolu- tional neural network for real-time pedestrian traffic lights and zebra crossing recognition for the visually impaired. In Computer Analysis of Images and Patterns: 18th Interna- tional Conference, CAIP 2019, Salerno, Italy, September 3–5, 2019, Proceedings, Part I 18, pages 259–270. Springer, 2019. 1

    2019

  34. [34]

    Street crossing aid using light-weight cnns for the visually impaired

    Samuel Yu, Heon Lee, and Junghoon Kim. Street crossing aid using light-weight cnns for the visually impaired. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019. 1

    2019

  35. [35]

    Minicpm-v 4.5: Cooking effi- cient mllms via architecture, data, and training recipe.arXiv preprint arXiv:2509.18154, 2025

    Tianyu Yu, Zefan Wang, Chongyi Wang, Fuwei Huang, Wen- shuo Ma, Zhihui He, Tianchi Cai, Weize Chen, Yuxiang Huang, Yuanqian Zhao, et al. Minicpm-v 4.5: Cooking effi- cient mllms via architecture, data, and training recipe.arXiv preprint arXiv:2509.18154, 2025. 7, 8, 3, 4

    Pith/arXiv arXiv 2025

  36. [36]

    Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490, 2023

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490, 2023. 10

    Pith/arXiv arXiv 2023

  37. [37]

    Walkvlm: Aid visually impaired people walking by vision language model.arXiv preprint arXiv:2412.20903, 2024

    Zhiqiang Yuan, Ting Zhang, Ying Deng, Jiapei Zhang, Yeshuang Zhu, Zexi Jia, Jie Zhou, and Jinchao Zhang. Walkvlm: Aid visually impaired people walking by vision language model.arXiv preprint arXiv:2412.20903, 2024. 1, 3, 4

    Pith/arXiv arXiv 2024

  38. [38]

    Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024. 7, 8

    Pith/arXiv arXiv 2024

  39. [39]

    Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024. 1, 3 10 VIABench: A Comprehensive Video Benchmark Collected from Blind Individuals for Visual Impairment Assistance Supplementary...

    Pith/arXiv arXiv 2024

  40. [40]

    Evaluation Settings Proactive Reminder.Offline models are evaluated using TPAD, while online models are tested with their default in- ference interfaces

    More Details of Evaluation 7.1. Evaluation Settings Proactive Reminder.Offline models are evaluated using TPAD, while online models are tested with their default in- ference interfaces. For closed-source models, we use single- frame inference according to their API constraints. VQA.For each question, we extract the video segment from the beginning of the ...

  41. [41]

    Additional Results 8.1. Proactive Reminder In theProactive Remindertask, the problem is inherently a two-stage end-to-end process.Stage 1is formulated as a frame-level binary classification task: given a sequence of video frames, the model predicts for each frame whether it should trigger a reminder. A prediction is counted as a true positive (TP) when th...

  42. [42]

    Is there a tactile paving nearby?

    and Kimi-VL-A3B [ 22]. This ordering reflects the ben- efits of increased model capacity as well as the advantages conferred by stronger instruction-tuned language generation. Across sub-tasks, the weakest performance is observed on Direction Deviation (DD). This task requires continuously tracking the user’s walking direction and issuing timely alerts wh...

  43. [43]

    shows the remaining time

    and Qwen2.5-VL-7B [2] tend to respond as if assisting a sighted user. Their answers simply point out that the control panel “shows the remaining time” or that it is “visible on top of the machine,” implicitly assuming that the user can see the screen. This mismatch reveals a deeper limitation: the models do not truly internalize the user’s identity as a b...

  44. [44]

    visible,

    read the wrong number) or providing actionable camera-adjustment guidance—open-source models such as InternVL3.5-8B [27] and Qwen2.5-VL-7B [2] assume a sighted user. Their responses simply state that the remaining-time display is “visible,” revealing a failure to adapt instructions to non-visual interaction. 8.6. Example Responses of Online Models As disc...

  45. [45]

    VIABench 9.1. Overview Our goal is to leverage multimodal large language models (MLLMs) as assistive agents for blind individuals, enabling more independent and safe navigation in real-world environ- ments. We envision deploying these models in smart glasses or smartphones to realize a Jarvis-like real-time visual assis- tant. Beyond proactively issuing a...

  46. [46]

    A wall blocks the end of the tactile path ahead

    Data Collection Details To rigorously and fairly evaluate MLLMs in real-world blind assistance, the quality of data collection and annotation is paramount. VIABench is built upon a carefully designed pipeline that prioritizes the authenticity, richness, and pre- cision of first-person video annotations. This section out- lines our data sourcing, annotatio...

  47. [47]

    Limitations Despite the strengths of VIABench in promoting real-time, vision-based assistive intelligence, several limitations re- main. First, although we reviewed extensive real-world blind- user videos to identify recurring assistive needs and distilled them into three major task families, the space of real-world visual assistance is fundamentally open...

  48. [48]

    Prompt Templates 12.1. Inference Prompts We provide here the prompt templates used across all exper- iments, covering proactive reminder sub-tasks, navigation- related queries, and VQA/VGI-style interactions. These templates define the expected response format and ensure consistency across different evaluation settings. The corre- sponding prompt designs ...

  49. [49]

    Wait, red

    Task Examples This section provides the task definitions, visual examples, and annotation examples for each task in VIABench in Fig- ure 15. 13 Figure 14. Detailed prompt templates for scoring all Proactive Reminder sub-tasks. 14 15 Figure 15. Task examples of Proactive Reminder [OA] Obstacle Alert Task Definition: Detect and describe nearby obstacles wit...