Pith. sign in

REVIEW 4 major objections 4 minor 47 references

This paper introduces TSHA, an 80k-pair benchmark for indoor safety hazard assessment, and claims that current vision-language models score poorly on it—while training on it raises scores by up to 18 points and improves general visual reaso

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 17:00 UTC pith:TNJMP23M

load-bearing objection TSHA is a genuinely large multi-source safety-hazard benchmark with a substantial evaluation, but the test set includes training samples and the GPT-4o labeler/evaluator loop inflates the headline gains; per-subset results and a release are needed before the +18.3 claim carries weight. the 4 major comments →

arxiv 2603.29759 v3 pith:TNJMP23M submitted 2026-03-31 cs.CV cs.AI

TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarios

Qiucheng Yu , Ruijie Xu , Mingang Chen , Jianfeng Dong , Xin Tan This is my paper
classification cs.CV cs.AI
keywords safety hazard assessmentvision-language modelsindoor safety benchmarkmultimodal question answeringAI-generated test datareinforcement learning fine-tuningbenchmark constructiondomain generalization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that prior safety-hazard benchmarks are built on synthetic simulator data and simplified tasks, so they do not reflect real homes. It introduces TSHA, a large benchmark of over 80,000 question-answer pairs assembled from real photos, internet frames, AI-generated images, and newly captured footage, plus a hard test set of videos and panoramic images with multiple simultaneous hazards. Evaluating 23 vision-language models, it finds that even the best closed-source model averages 66 out of 100, and open-ended hazard identification is the weakest spot for every model. Training on TSHA's training set improves a small 3-billion-parameter model by 18.3 points on the TSHA test set and gives small gains across five general vision-language benchmarks. A sympathetic reader would take the paper's central claim to be that this resource provides both a sharper evaluation tool and a transferable training signal for making vision-language models more safety-aware.

Core claim

On the paper's own terms, the central discovery is that a large, realistic, diverse dataset can both expose and partially close the gap between current vision-language models and competent indoor safety assessment. The test set was selected for difficulty: the best closed-source model averages 66 out of 100, open-source models vary from about 33 to 82, and open-ended hazard enumeration is where all models fail even when multiple-choice accuracy looks high. Fine-tuning with reinforcement learning on TSHA's 80,102 training pairs lifts a small 3B model by 18.3 points on the test set, and the same training produces consistent, modest gains on five unrelated general benchmarks, which the authors

What carries the argument

The load-bearing object is TSHA (Trustworthy Safety Hazards Assessment): a collection of 80,102 training question-answer pairs and 1,707 test pairs. Training data come from four sources—existing indoor image datasets, frames from internet videos, AI-generated images, and newly captured home/work videos—and the test set adds AI-generated videos and panoramic images containing co-occurring hazards. Each image is annotated through a four-stage pipeline: objective scene description, preliminary hazard screening, evidence-based final hazard identification with confidence levels, and human verification of a 10% random sample, all driven by the proprietary model GPT-4o. Evaluation uses accuracy for

Load-bearing premise

The entire benchmark rests on the assumption that the GPT-4o-generated question-answer annotations are correct descriptions of real hazards, but only 10% of samples were checked by humans.

What would settle it

Take a random 200-sample subset of the TSHA test set and have two independent safety experts annotate the hazards from scratch, without seeing the GPT-4o labels; if expert agreement with the benchmark's ground truth is low (say, below 80%), the benchmark would be measuring agreement with GPT-4o rather than real safety. In parallel, train a small model on TSHA and evaluate it on newly collected photos of cluttered real homes with expert labels; a large drop relative to TSHA test scores would qualify the paper's generalization claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If TSHA's results hold, no current vision-language model is ready for unsupervised home-safety inspection; average scores near 60–66 hide much worse performance on open-ended hazard reporting.
  • Training on TSHA's mixed real/AI data transfers to general vision-language benchmarks, suggesting that safety-hazard data teaches transferable perceptual reasoning, not just test-specific answers.
  • Choice-question accuracy can be 80–90% while open-ended accuracy lingers near 60%, so evaluations built only on multiple choice will overstate a model's real safety competence; open-ended QA should be the primary metric.
  • The four data sources are complementary: ablations show each source helps on its own and the combination yields the largest gains, implying that imagery diversity is a design choice worth copying.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because only 10% of annotations were human-verified, the benchmark's accuracy ceiling is set by GPT-4o's judgment; a human re-annotation study could reveal whether TSHA measures objective safety or agreement with one model.
  • Editorial inference: the test set includes a 'carefully selected subset' from the training distribution, so part of the +18.3 gain may reflect distribution overlap rather than generalization; the cleanest test of transfer would be evaluation on freshly collected real-world images with expert labels.
  • Editorial inference: the LLM-as-judge scoring weights conciseness and coherence alongside accuracy, so some of the training gain could come from stylistic alignment with the judge; correlating judge scores with expert human ratings on the same responses would separate style from substance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces TSHA, a large benchmark for indoor safety hazard assessment, with training QA pairs sourced from existing indoor datasets, internet images, AIGC images, and newly captured images, plus a test set that adds Sora2 videos and Hunyuan panoramic images. The authors evaluate 23 VLMs, report that current models score poorly, and show that GRPO fine-tuning on TSHA improves Qwen2.5-VL-3B by up to +18.3 points on the TSHA test set, with smaller gains on general benchmarks. The benchmark is intended both as an evaluation suite and as a training resource.

Significance. TSHA addresses a genuine gap: prior home-safety benchmarks are small, often simulator-based, and restricted in hazard types. The scale of the collected data, the diversity of sources, the inclusion of video and panoramic test samples, and the systematic evaluation across 23 models are all strengths. If the validity concerns were resolved, TSHA could be a useful community resource. The paper also includes a human evaluation and ablations over data sources, which are commendable. However, the central empirical claims—that current VLMs are notably weak on this task and that TSHA training yields large generalization gains—are not currently supported because of train/test overlap, GPT-4o-based label generation and evaluation, and unresolved data-scale inconsistencies.

major comments (4)
  1. [§3.1, Table 2] The test set description in §3.1 says it includes 'a carefully selected subset of the training data', and the test distribution in Fig. 3b includes samples from the same four sources used in training (existing datasets, newly captured images, internet images, AIGC images). The abstract instead says 'subset from the training distribution'. If the test set literally contains training QA pairs, the headline +18.3 gain for Qwen2.5-VL-3B in Table 2 could reflect memorization rather than generalization. The paper reports only aggregate test scores and never separates the overlapping subset from the novel Sora2/Hunyuan portions. Please either remove overlapping samples from the test set or report per-subset results, and specify the exact degree of overlap.
  2. [§3.3, §4.1, Table 5] The benchmark's ground truth is generated by GPT-4o with a chain-of-thought pipeline, and human verification covers only 10% of the samples (§3.3). The main automated evaluator for open-ended QA is ChatGPT-4o (§4.1, Eq. 1). Thus the reported scores partly measure agreement with GPT-4o's hazard judgments. Table 5 shows that the gains under GPT-4o evaluation (+10.4 for the 3B model) shrink to +2.4 under human evaluation; similarly for 7B (+3.2 vs +2.1) and 32B (+1.6 vs +0.6). This suggests the headline improvements are inflated by the shared model. Please provide larger-scale human validation of the ground-truth labels and report the primary results under human evaluation or another independent evaluator, together with human-GPT-4o agreement.
  3. [Abstract, §3.2, Table 1] The total number of QA pairs is inconsistent: the abstract states 66,668 validated QA pairs including 64,961 training pairs; the full-text abstract and Table 1 state 81,809 training samples; §3.2 states the final training set comprises 80,102 QA pairs. The test set is consistently 1,707. These discrepancies are not explained and make it unclear which exact dataset was used for training and for the +18.3 result. Please reconcile all numbers and report the exact training/test composition used in the experiments.
  4. [§4.3, Table 3] The text states that TSHA-trained models achieve 'statistically significant performance improvements across all general vision-language benchmark datasets.' Table 3 shows average gains of +1.6, +1.2, and +0.4 for the 3B, 7B, and 72B models, with negative changes for the 72B model on BLINK (-2.2) and MMStar (-1.1). No significance tests, confidence intervals, or multiple-run statistics are provided. Please either provide statistical support for this claim or soften the wording to describe the observed trends.
minor comments (4)
  1. [§3.3] The text first says 10% of samples were randomly selected for manual inspection, then later says 'every generated example undergoes manual verification'. These statements conflict; please clarify the verification protocol.
  2. [Fig. 3] The axis labels and legends in Fig. 3 appear as garbled character codes and are not interpretable. Please regenerate the figure with readable text.
  3. [Abstract, §4.2] The model count is inconsistent: the abstract says 22 VLMs, while §4.2 and Table 2 evaluate 23. The text also says InternVL2.5-26B led with 82.8, but Table 2 lists its average score as 80.8. Please correct these numbers.
  4. [Table 2] The Minicpm-4B row appears twice in the same block, which makes it hard to distinguish the baseline from the TSHA-trained variant. Please reformat the table.

Circularity Check

2 steps flagged

Headline +18.3 gain is partly circular: the TSHA test set includes a subset of the training data, and the GPT-4o labeler doubles as the ChatGPT-4o grader.

specific steps
  1. fitted input called prediction [Section 3.1 (TSHA Test Set) and Section 4.2 / Table 2 (Main Results)]
    "[Abstract] models trained on the TSHA training set achieve a significant performance improvement of up to +18.3 points on the TSHA test set. [Sec. 3.1] our test set introduces greater complexity by integrating three complementary components: video sequences, panoramic images, and a carefully selected subset of the training data."

    The claimed +18.3 gain is measured on a test set that explicitly contains a subset of the training data. For that subset, the model is re-tested on examples it was trained on, so the gain reflects memorization or in-distribution fit rather than generalization. Table 2 reports only aggregate TSHA scores, with no separate results for the training-subset portion versus the novel Sora2-video/panoramic-image portion, so the headline improvement cannot be decomposed. The 'enhanced generalizability' prediction is therefore partly the same as the training input by construction.

  2. self definitional [Section 3.3 (Multimodal Dialogue Generation) and Section 4.1 (Experimental Setup, Metrics)]
    "[Sec. 3.3] We employ the GPT-4o [18] model to generate contextual, multi-turn dialogues through a carefully designed chain-of-thought reasoning process. [Sec. 4.1] Specifically, we employ ChatGPT-4o [18] to assess model responses against reference answers across three dimensions: (1) Accuracy..."

    The training targets and the automated grading reference are produced by the same model (GPT-4o / ChatGPT-4o). GRPO training on GPT-4o-generated QA pairs optimizes the model to align with GPT-4o's hazard judgments, and the ChatGPT-4o grader then rewards that same alignment. Thus the automated QA score is, in part, a self-agreement score rather than an independent measure of safety-hazard competence. The paper's own Table 5 shows the headline gain for Qwen2.5-VL-3B shrinks from +10.4 under GPT-4o to +2.4 under human evaluation, confirming that the loop inflates the measured improvement.

full rationale

The central improvement claim of TSHA is not self-contained. The test set deliberately includes a subset of the training data (Sec. 3.1), so the reported +18.3-point gain on the TSHA test set conflates memorization with generalization; the paper never reports the training-subset/novel split needed to separate them. Additionally, the benchmark's ground truth was generated by GPT-4o (Sec. 3.3) and the automated evaluator is ChatGPT-4o (Sec. 4.1), so the headline automated scores partly measure agreement with GPT-4o rather than an independent safety ground truth; the small human-evaluated gains in Table 5 confirm the inflation. These are the two load-bearing circular/confounded steps. The paper is not a pure self-citation chain, and its external-benchmark results (Blink, MMStar, AI2D, MUIR, SEED) plus the construction of a large image corpus provide independent content, so a score of 6 (partial circularity) rather than 8-10 is appropriate. No self-citation uniqueness theorem or renaming pattern is present.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The benchmark introduces no new physical or conceptual entities; TSHA is a dataset resource. The main hidden costs are the trust assumptions placed in GPT-4o as labeler, the 10% human verification rate, and the hand-chosen evaluation weights.

free parameters (1)
  • QA evaluation weights = 0.7 / 0.2 / 0.1
    Equation (1) sets S_QA = 0.7*S_acc + 0.2*S_con + 0.1*S_coh; the weights are chosen by the authors based on 'practical significance', not derived from data. They directly determine all reported QA and overall scores.
axioms (4)
  • domain assumption GPT-4o-generated descriptions and hazard labels are sufficiently accurate to serve as ground truth.
    Section 3.3 relies on GPT-4o for scene description, preliminary hazard identification, and final hazard identification, with only 10% of samples receiving human verification.
  • domain assumption A 10% random human sample is enough to certify the remaining 90% of the dataset.
    Section 3.3 treats '10% of the total samples are randomly selected for manual inspection' as sufficient quality validation for the full dataset.
  • domain assumption LLM evaluators (ChatGPT-4o, Claude-3.7-sonnet, Deepseek-r1) produce valid numeric scores for open-ended safety QA.
    Section 4.1 uses LLM judges as the primary metric; their scores are averaged across dimensions but are not validated against a gold-standard human rating at scale.
  • domain assumption Images from NYU v2, MIT Indoor, internet videos, AIGC, and volunteer captures form a representative distribution of real home hazards.
    Section 3.1 assumes that these sources cover the broad 'spectrum of real-world safety concerns' without a formal sampling or coverage analysis.

pith-pipeline@v1.3.0-alltime-deepseek · 18313 in / 17684 out tokens · 152806 ms · 2026-08-02T17:00:28.227049+00:00 · methodology

0 comments
read the original abstract

Recent advances in vision-language models (VLMs) have accelerated their application to indoor safety hazards assessment. However, existing benchmarks suffer from three fundamental limitations: (1) heavy reliance on synthetic datasets constructed via simulation software, creating a significant domain gap with real-world environments; (2) oversimplified safety tasks with artificial constraints on hazard and scene types, thereby limiting model generalization; and (3) absence of rigorous evaluation protocols to thoroughly assess model capabilities in complex home safety scenarios. To address these challenges, we introduce TSHA (\textbf{T}rustworthy \textbf{S}afety \textbf{H}azards \textbf{A}ssessment), a comprehensive benchmark comprising 66,668 validated question-answer pairs, including 64,961 carefully curated training QA pairs drawn from existing indoor datasets, internet frames/images, AIGC images, newly captured images, and Hunyuan panoramic images. This benchmark also includes a highly challenging test set with 1,707 QA pairs, comprising not only a carefully selected subset from the training distribution but also newly added Sora-generated videos and Hunyuan panoramic images containing multiple safety hazards, used to evaluate the model's robustness in complex safety scenarios. Extensive experiments on 22 popular VLMs demonstrate that current VLMs lack robust capabilities for safety hazard assessment. Importantly, models trained on the TSHA training set achieve a significant performance improvement of up to +18.3 points on the TSHA test set and also exhibit enhanced generalizability across other benchmarks, underscoring the substantial contribution and importance of the TSHA benchmark.

Figures

Figures reproduced from arXiv: 2603.29759 by Jianfeng Dong, Mingang Chen, Qiucheng Yu, Ruijie Xu, Xin Tan.

Figure 1
Figure 1. Figure 1: Overview of image, video and question samples of TSHA datasets. 3 TSHA In this section, we introduce TSHA, a comprehensive benchmark specifically designed to evaluate the ability of vision-language models (VLMs) to assess safety hazards. Sec. 3.1 details the data sources of the training and test sets, as well as the rigorous filtering and quality control procedures implemented to ensure dataset reliability… view at source ↗
Figure 2
Figure 2. Figure 2: Data Generation Pipeline. The collected images and videos from the data source undergo a four-stage annotation process: scene description, preliminary hazard identification, final hazard verification, and human verification. the distribution of authentic safety hazards. For Internet images, we carefully selected and downloaded 63 high-quality videos of everyday home life from plat￾forms such as YouTube and… view at source ↗
Figure 3
Figure 3. Figure 3: Dataset statistics. Fig. 3a demonstrates the training data distribution, Fig. 3b shows the test data distribution and Fig. 3c indicates the Top-15 safety hazards of TSHA. while explicitly prohibiting any safety-related judgments. This step establishes a factual, assumption-free foundation for subsequent analysis. Building on this description, the Preliminary Hazards Identification stage performs an initial… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 28 linked inside Pith

  1. [1]

    Advances in neural information processing systems35, 23716– 23736 (2022)

    Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Men- sch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems35, 23716– 23736 (2022)

    2022

  2. [2]

    In: Submitted to The Fourteenth Inter- national Conference on Learning Representations (2025),https://openreview

    Anonymous: Robotrust: Evaluating the interaction trustworthiness of multi-modal large language models in embodied agents. In: Submitted to The Fourteenth Inter- national Conference on Learning Representations (2025),https://openreview. net/forum?id=q6wVgopVnq, under review

    2025

  3. [3]

    anthropic.: Introducing the next generation of claude

  4. [4]

    5-vl technical report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    Pith/arXiv arXiv 2025

  5. [5]

    OpenAI Blog1(8), 1 (2024)

    Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., et al.: Video generation models as world simulators. OpenAI Blog1(8), 1 (2024)

    2024

  6. [6]

    Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., Zhao, F.: Are we on the right way for evaluating large vision-language models? (2024),https://arxiv.org/abs/2403.20330

    Pith/arXiv arXiv 2024

  7. [7]

    arXiv preprint arXiv:2412.05271 (2024)

    Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al.: Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 (2024)

    Pith/arXiv arXiv 2024

  8. [8]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 24185–24198 (2024)

    2024

  9. [9]

    arXiv preprint arXiv:2507.06261 (2025)

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al.: Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025)

    Pith/arXiv arXiv 2025

  10. [10]

    Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., Lin, D., Chen, K.: Vlmevalkit: An open-source toolkit for evaluating large multi-modality models (2024),https://arxiv.org/abs/2407. 11691

    2024

  11. [11]

    In: European Conference on Computer Vision

    Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., Roth, D., Smith, N.A., Ma, W.C., Krishna, R.: Blink: Multimodal large language models can see but not per- ceive. In: European Conference on Computer Vision. pp. 148–166. Springer (2024)

    2024

  12. [12]

    arXiv preprint arXiv:2509.23690 (2025)

    Gao, S., Yao, J., Wen, H., Guo, Y., Liu, Z., Huang, H.: Homesafebench: A bench- mark for embodied vision-language models in free-exploration home safety inspec- tion. arXiv preprint arXiv:2509.23690 (2025)

    arXiv 2025

  13. [13]

    arXiv preprint arXiv:2503.07703 (2025)

    Gong, L., Hou, X., Li, F., Li, L., Lian, X., Liu, F., Liu, L., Liu, W., Lu, W., Shi, Y., et al.: Seedream 2.0: A native chinese-english bilingual image generation foundation model. arXiv preprint arXiv:2503.07703 (2025)

    Pith/arXiv arXiv 2025

  14. [14]

    5-vl technical report

    Guo, D., Wu, F., Zhu, F., Leng, F., Shi, G., Chen, H., Fan, H., Wang, J., Jiang, J., Wang, J., et al.: Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062 (2025)

    Pith/arXiv arXiv 2025

  15. [15]

    arXiv preprint arXiv:2410.14141 (2024) Title Suppressed Due to Excessive Length 17

    Hassan, S., Chung, H.Y., Tan, X.Z., Alikhani, M.: Coherence-driven multi- modal safety dialogue with active learning for embodied agents. arXiv preprint arXiv:2410.14141 (2024) Title Suppressed Due to Excessive Length 17

    Pith/arXiv arXiv 2024

  16. [16]

    arXiv preprint arXiv:2506.04225 (2025)

    Huang, T., Zheng, W., Wang, T., Liu, Y., Wang, Z., Wu, J., Jiang, J., Li, H., Lau, R.W., Zuo, W., Guo, C.: Voyager: Long-range and world-consistent video diffusion for explorable 3d scene generation. arXiv preprint arXiv:2506.04225 (2025)

    Pith/arXiv arXiv 2025

  17. [17]

    arXiv preprint arXiv:2504.14650 (2025)

    Huang, Y., Ding, L., Tang, Z., Wang, T., Lin, X., Zhang, W., Ma, M., Zhang, Y.: A framework for benchmarking and aligning task-planning safety in llm-based embodied agents. arXiv preprint arXiv:2504.14650 (2025)

    Pith/arXiv arXiv 2025

  18. [18]

    arXiv preprint arXiv:2410.21276 (2024)

    Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Os- trow, A., Welihinda, A., Hayes, A., Radford, A., et al.: Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024)

    Pith/arXiv arXiv 2024

  19. [19]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Ji, Y., Tan, H., Shi, J., Hao, X., Zhang, Y., Zhang, H., Wang, P., Zhao, M., Mu, Y., An, P., et al.: Robobrain: A unified brain model for robotic manipulation from abstract to concrete. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 1724–1734 (2025)

    2025

  20. [20]

    In: European conference on computer vision

    Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., Farhadi, A.: A di- agram is worth a dozen images. In: European conference on computer vision. pp. 235–251. Springer (2016)

    2016

  21. [21]

    Li, B., Ge, Y., Ge, Y., Wang, G., Wang, R., Zhang, R., Shan, Y.: Seed-bench- 2: Benchmarking multimodal large language models (2023),https://arxiv.org/ abs/2311.17092

    Pith/arXiv arXiv 2023

  22. [22]

    Advances in neural information processing systems36, 34892–34916 (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36, 34892–34916 (2023)

    2023

  23. [23]

    arXiv preprint arXiv:2508.11918 (2025)

    Lou, Z., Xu, K., Zhou, Z., Xiong, R.: Explorevlm: Closed-loop robot exploration task planning with vision-language models. arXiv preprint arXiv:2508.11918 (2025)

    Pith/arXiv arXiv 2025

  24. [24]

    arXiv preprint arXiv:2406.08451 (2024)

    Lu, Q., Shao, W., Liu, Z., Meng, F., Li, B., Chen, B., Huang, S., Zhang, K., Qiao, Y., Luo, P.: Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.08451 (2024)

    Pith/arXiv arXiv 2024

  25. [25]

    arXiv:2405.20797 (2024)

    Lu, S., Li, Y., Chen, Q.G., Xu, Z., Luo, W., Zhang, K., Ye, H.J.: Ovis: Struc- tural embedding alignment for multimodal large language model. arXiv:2405.20797 (2024)

    Pith/arXiv arXiv 2024

  26. [26]

    arXiv:2508.11737 (2025)

    Lu, S., Li, Y., Xia, Y., Hu, Y., Zhao, S., Ma, Y., Wei, Z., Li, Y., Duan, L., Zhao, J., Han, Y., Li, H., Chen, W., Tang, J., Hou, C., Du, Z., Zhou, T., Zhang, W., Ding, H., Li, J., Li, W., Hu, G., Gu, Y., Yang, S., Wang, J., Sun, H., Wang, Y., Sun, H., Huang, J., He, Y., Shi, S., Zhang, W., Zheng, G., Jiang, J., Gao, S., Wu, Y.F., Chen, S., Chen, Y., Chen...

    Pith/arXiv arXiv 2025

  27. [27]

    Image and Vision Computing151, 105259 (2024)

    Luo, H., Zeng, Y., Yang, L., Chen, K., Shen, Z., Lv, F.: Vlai: Exploration and exploitation based on visual-language aligned information for robotic object goal navigation. Image and Vision Computing151, 105259 (2024)

    2024

  28. [28]

    arXiv preprint arXiv:2506.07900 (2025)

    MiniCPM, T.: Minicpm4: Ultra-efficient llms on end devices. arXiv preprint arXiv:2506.07900 (2025)

    Pith/arXiv arXiv 2025

  29. [29]

    don’t forget to put the milk back!

    Mullen, J.F., Goyal, P., Piramuthu, R., Johnston, M., Manocha, D., Ghanadan, R.: “don’t forget to put the milk back!” dataset for enabling embodied agents to detect anomalous situations. IEEE Robotics and Automation Letters9(10), 9087–9094 (2024)

    2024

  30. [30]

    In: 2009 IEEE conference on computer vision and pattern recognition

    Quattoni, A., Torralba, A.: Recognizing indoor scenes. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 413–420. IEEE (2009)

    2009

  31. [31]

    arXiv preprint arXiv:2506.10910 (2025) 18 Authors Suppressed Due to Excessive Length

    Rastogi, A., Jiang, A.Q., Lo, A., Berrada, G., Lample, G., Rute, J., Barmentlo, J., Yadav, K., Khandelwal, K., Chandu, K.R., et al.: Magistral. arXiv preprint arXiv:2506.10910 (2025) 18 Authors Suppressed Due to Excessive Length

    Pith/arXiv arXiv 2025

  32. [32]

    arXiv preprint arXiv:2402.03300 (2024)

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024)

    Pith/arXiv arXiv 2024

  33. [33]

    In: European conference on computer vision

    Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from rgbd images. In: European conference on computer vision. pp. 746–

  34. [34]

    In: European conference on computer vision

    Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., Beißwenger, J., Luo, P., Geiger, A., Li, H.: Drivelm: Driving with graph visual question answering. In: European conference on computer vision. pp. 256–274. Springer (2024)

    2024

  35. [35]

    arXiv preprint arXiv:2411.02265 (2024)

    Sun, X., Chen, Y., Huang, Y., Xie, R., Zhu, J., Zhang, K., Li, S., Yang, Z., Han, J., Shu, X., et al.: Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent. arXiv preprint arXiv:2411.02265 (2024)

    Pith/arXiv arXiv 2024

  36. [36]

    arXiv preprint arXiv:2503.19786 (2025)

    Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., et al.: Gemma 3 technical report. arXiv preprint arXiv:2503.19786 (2025)

    Pith/arXiv arXiv 2025

  37. [37]

    Wang, F., Fu, X., Huang, J.Y., Li, Z., Liu, Q., Liu, X., Ma, M.D., Xu, N., Zhou, W., Zhang, K., Yan, T.L., Mo, W.J., Liu, H.H., Lu, P., Li, C., Xiao, C., Chang, K.W., Roth, D., Zhang, S., Poon, H., Chen, M.: Muirbench: A comprehensive benchmark for robust multi-image understanding (2024),https://arxiv.org/ abs/2406.09411

    Pith/arXiv arXiv 2024

  38. [38]

    arXiv preprint arXiv:2510.03666 (2025)

    Wu, J., Wu, S., Ma, Y., Yu, G., Xu, H., Zheng, L., Duan, J.: Monitorvlm: A vi- sion language framework for safety violation detection in mining operations. arXiv preprint arXiv:2510.03666 (2025)

    arXiv 2025

  39. [39]

    Xiaomi, L.C.T.: Mimo-vl technical report (2025),https://arxiv.org/abs/2506. 03569

    2025

  40. [40]

    arXiv preprint arXiv:2412.14446 (2024)

    Xu, Y., Hu, Y., Zhang, Z., Meyer, G.P., Mustikovela, S.K., Srinivasa, S., Wolff, E.M., Huang,X.: Vlm-ad: End-to-endautonomous drivingthrough vision-language model supervision. arXiv preprint arXiv:2412.14446 (2024)

    Pith/arXiv arXiv 2024

  41. [41]

    arXiv preprint arXiv:2412.13178 (2024)

    Yin, S., Pang, X., Ding, Y., Chen, M., Bi, Y., Xiong, Y., Huang, W., Xiang, Z., Shao, J., Chen, S.: Safeagentbench: A benchmark for safe task planning of embodied llm agents. arXiv preprint arXiv:2412.13178 (2024)

    arXiv 2024

  42. [42]

    arXiv preprint arXiv:2509.16445 (2025)

    Yokoyama, N., Ha, S.: Film-nav: Efficient and generalizable navigation via vlm fine-tuning. arXiv preprint arXiv:2509.16445 (2025)

    Pith/arXiv arXiv 2025

  43. [43]

    arXiv preprint arXiv:2408.09251 (2024)

    You, J., Shi, H., Jiang, Z., Huang, Z., Gan, R., Wu, K., Cheng, X., Li, X., Ran, B.: V2x-vlm: End-to-end v2x cooperative autonomous driving through large vision- language models. arXiv preprint arXiv:2408.09251 (2024)

    Pith/arXiv arXiv 2024

  44. [44]

    arXiv preprint arXiv:2502.14254 (2025)

    Zhang, L., Liu, Y., Zhang, Z., Aghaei, M., Hu, Y., Gu, H., Alomrani, M.A., Bravo, D.G.A., Karimi, R., Hamidizadeh, A., et al.: Mem2ego: Empowering vision- language models with global-to-ego memory for long-horizon embodied navigation. arXiv preprint arXiv:2502.14254 (2025)

    Pith/arXiv arXiv 2025

  45. [45]

    Zhao, Y., Huang, J., Hu, J., Wang, X., Mao, Y., Zhang, D., Jiang, Z., Wu, Z., Ai, B., Wang, A., Zhou, W., Chen, Y.: Swift:a scalable lightweight infrastructure for fine-tuning (2024),https://arxiv.org/abs/2408.05517

    Pith/arXiv arXiv 2024

  46. [46]

    arXiv preprint arXiv:2401.12975 (2024)

    Zhou,Q.,Chen,S.,Wang,Y.,Xu,H.,Du,W.,Zhang,H.,Du,Y.,Tenenbaum,J.B., Gan, C.: Hazard challenge: Embodied decision making in dynamically changing environments. arXiv preprint arXiv:2401.12975 (2024)

    Pith/arXiv arXiv 2024

  47. [47]

    arXiv preprint arXiv:2504.10479 (2025)

    Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., et al.: Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 (2025)

    Pith/arXiv arXiv 2025