Pith. sign in

REVIEW 2 major objections 6 minor 1 cited by

SteelBench: Evaluating Vision-Language Models in Real-World Industrial Environments

T0 review · 2 major / 6 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read On real steel-plant CCTV, the best vision-language model hits only 42.6% action accuracy against an 84.6% human reference, and correct actions still produce wrong safety judgments 37–58% of the time.

desk verdict Real-plant industrial VLM benchmark with a quantified provenance audit; the 42pp human gap and 17pp same-family inflation are the numbers that matter. read the letter →

arxiv 2607.05264 v1 pith:AKYMRSXN submitted 2026-07-06 cs.CV

Suryanarayana Reddy Yarrabothula , Manisha Chawla , Kunal Sinha , Gagan Raj Gupta , Sashank Lekkala , Ashirvadhan Dosapati , Saikamal Nannuri , Katragadda Ajay RamaSwamy Chowdary Gowtham This is my paper · ORCID
classification cs.CV
keywords vision-languagemodelsindustrialsurveillanceactionrecognitionsafetyreasoningannotationprovenancebenchmarkPPEcomplianceCCTV
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SteelBench argues that existing video benchmarks do not test vision-language models under the conditions of real industrial surveillance: distant workers, dust, steam, glare, occlusion, and overlapping activities. The authors release 1,345 densely annotated clips from an operating steel plant, each carrying per-worker actions, PPE status, spatial context, and safety-rule labels. Because the labels themselves were built with model help, they also introduce a provenance-aware audit that measures how much VLM pre-fills shape final ground truth. The audit shows that unaudited same-family labels can inflate measured accuracy by up to 17 percentage points. Across nine models, recognition, robustness, calibration, and safety reasoning fail independently, and no model passes more than two of five deployment-readiness checks. The paper’s claim is that reliable industrial activity understanding requires provenance-aware, failure-mode-specific evaluation rather than a single accuracy leaderboard.

What carries the argument

The provenance-aware audit protocol: a three-level procedure that measures label influence from VLM pre-fills (productive vs harmful anchors and overrides), re-scores the same model predictions against blind, proper-chain, and VLM-sourced ground truth, and reports a human reference from expert-reviewed labels.

What would settle it

Re-label a blind, model-free subset of the same clips with independent experts and re-run the nine models: if the human–model gap collapses or the provenance inflation shrinks dramatically, the central performance claim does not hold.

Watch

Extended reading notes

Core claim

On real operational CCTV, the strongest of nine vision-language models reaches only 42.6% action accuracy versus an 84.6% human reference; even when the action is correct, 37–58% of safety judgments remain wrong; and unaudited VLM-sourced ground truth can inflate same-family accuracy by up to 17 percentage points. No model passes more than two of five diagnostic deployment checks.

Load-bearing premise

The expert-verified labels used as the gold standard remain independent enough of the original model pre-fills that the large human–model gap and the reported accuracy inflation are trustworthy.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. SteelBench is a diagnostic benchmark for vision-language models on real industrial CCTV from an operational integrated steel plant. It provides 1,345 densely annotated clips (from 149 hours / 10,024 candidates) with per-worker actions (25 classes), PPE, spatial context, visibility, and safety-rule labels under a 2-layer schema. The central methodological contribution is a three-level provenance-aware audit of model-assisted annotation (label influence via CLR/CR and anchoring bias; GT provenance sensitivity across blind / proper-chain / VLM-sourced slices; human reference). Evaluating nine VLMs, the paper reports a large human–model gap (best 42.6% action accuracy vs 84.6% human reference), up to ~17–40pp inflation under VLM-sourced GT for same-family evaluation, high compositional reasoning gaps (CRG 0.375–0.582 even on action-correct instances), fragmented robustness/calibration, and at most 2/5 DRS diagnostic checks passed. Ablations cover prompt variants, frame density, threshold sensitivity, and domain/site/condition breakdowns; data and code are released.

Significance. If the results hold, the paper is a substantial contribution on two fronts: (i) a realistic industrial surveillance evaluation surface that existing consumer, egocentric, and simulated industrial benchmarks do not provide, and (ii) an explicit, measurable protocol for auditing circularity in model-assisted benchmark construction—an issue widely acknowledged but rarely quantified. Strengths include multi-slice provenance experiments, exclusion of a rubber-stamping annotator via audit signals, human reference on 370 person-level pairs with κ, bootstrap CIs, prompt/frame ablations, DRS threshold sensitivity, and public release under CC-BY-NC-4.0 with code. The diagnostic framing (recognition vs robustness vs calibration vs safety reasoning) is more useful for deployment decisions than a single leaderboard score. Single-facility scope and residual anchoring are real limits but are disclosed and partially mitigated by internal domain diversity and quantified residual rates.

major comments (2)
  1. §5.1 Level 3 and Appendix D.7: The headline 84.6% human reference (and thus the 42pp gap) is measured on proper-chain pairs where Tier-1 annotators saw Qwen3-VL-235B pre-fills before expert verification. Blind-condition human accuracy against the expert is described as lower but is not reported as a primary number alongside 84.6%. Because residual harmful anchoring is 3.9% and the paper’s own Level-2 gradient shows large provenance effects, the main human–model gap claim would be stronger if blind human accuracy (and its n) were stated in §5.1 / Table 4 as a co-primary reference, with proper-chain retained as the operational annotation-workflow baseline.
  2. §4 and Appendix C.2.6–C.2.8 (DRS / DWA): DWA’s taxonomic distances (0 / 0.33 / 0.60 / 0.70–1.00) and the five DRS thresholds are load-bearing for the claim that no model is deployment-ready (≤2/5 checks). Threshold sensitivity (±10%) is analyzed and margins for CRG and DWA are large, which is good; however, the distance weights themselves are not justified against plant incident severity or officer ranking beyond narrative rationale. A short sensitivity table over alternative distance schemes (or collapsing to group-level accuracy) would show whether the DWA fail is robust or scheme-dependent, analogous to the existing threshold ablation.
minor comments (6)
  1. Abstract vs §5.1: Abstract says unaudited VLM-sourced GT can inflate same-family accuracy by “up to 17 percentage points,” while Table 4 / text report 37.2% → 57.4% → 77.7% (larger spans). Align the abstract figure with the exact contrast intended (e.g., proper-chain vs VLM-sourced ≈20pp, or blind vs VLM-sourced).
  2. Figure 4 / Table 6: Classes C4 and D3 have n<15 and are correctly excluded from per-class claims, but the heatmap still displays them without a clear visual marker; add a hatch or footnote so readers do not over-read those cells.
  3. Table 1 vs Appendix A.3: Layer-2 / Layer-1 clip counts differ slightly across places (805/540 vs 807/538). Harmonize final counts after annotator_10 exclusion.
  4. §2 and Figure 2: “Class balancing” in the abstract/curation text sits awkwardly next to the retained natural long-tail distribution; clarify that stratified sampling enforces minimum support where available rather than uniform class balance.
  5. Appendix E.2 V2: Free-form descriptions are mapped by GPT-4o-mini; state mapper agreement or a small human audit so V2 accuracy is not over-interpreted relative to V1/V3.
  6. Minor polish: arXiv id / preprint date consistency; expand first use of MAI/MAC/SA in the main text before Figure 1; fix occasional spacing (e.g., “CCTV ,”, “A V A”).

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circular derivation; residual VLM pre-fill influence is quantified by the paper's own provenance audit rather than hidden or used to force results.

full rationale

SteelBench is an empirical diagnostic benchmark paper, not a first-principles derivation. Its central claims (42.6% best-model action accuracy vs 84.6% human reference, CRG 0.375–0.582, ≤2/5 DRS, and up to 17 pp same-family inflation under VLM-sourced GT) are obtained by evaluating nine external VLMs on a newly constructed, multi-stage annotated dataset and reporting standard and custom metrics (accuracy, nAUDC, CRG, ECE, DRS). The annotation pipeline begins with Qwen3-VL-235B pre-fills, which creates a potential dependence, but the paper's explicit contribution is a three-level provenance-aware audit (CLR 92.8%, CR 6.9×, harmful anchoring 3.9% on 2 829 fields; blind/proper-chain/VLM-sourced accuracy gradient 37.2%→57.4%→77.7%; human reference 84.6% κ=0.82 on 370 proper-chain pairs) that measures and stratifies this influence rather than concealing it. Models are scored primarily against proper-chain and blind slices; a rubber-stamping annotator is excluded by the same audit. Self-citation of the authors' prior work [28] appears only in Related Work and is not load-bearing. DRS thresholds come from plant safety-officer input, not post-hoc fitting to produce failures. No equation, prediction, or uniqueness claim reduces by construction to its own inputs. The residual 3.9% harmful anchoring and ~6.6 pp universal anchored-GT inflation are disclosed limitations, not circular reductions of the headline gaps (which remain an order of magnitude larger). Score 1 reflects only this transparent residual dependence; the derivation chain itself is self-contained and non-circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 5 invented entities

The central empirical claims rest on a small set of design choices (8-frame sampling, DRS thresholds, expert-as-gold, single-plant sampling) and on newly defined diagnostic quantities (CRG, nAUDC, DRS, CLR/CR). No physical constants or free-form curve fits are involved; the free parameters are evaluation thresholds and protocol knobs. Invented entities are the benchmark and the audit/diagnostic metrics themselves.

free parameters (4)
  • DRS component thresholds (DWA≥0.80, nAUDC≥0.85, CRG≤0.20, safety recall≥0.90, F2-detect≥0.70)
    Chosen from plant safety-officer inputs; sensitivity to ±10–20% is analyzed but the exact cut-offs remain free design parameters that determine the 'no model passes >2/5' headline.
  • 8-frame sampling protocol
    Fixed proportional positions at 1080p; ablation shows architecture-dependent peaks, so the protocol is a free experimental choice that affects absolute accuracy.
  • Per-class support floor n=15
    Classes below this threshold are excluded from per-class claims; the cutoff is conventional rather than derived.
  • Person-detection trigger (YOLOv8n conf>0.25 for ≥5 frames)
    Controls which candidate clips enter the pool; deliberately permissive but still a free pipeline parameter.
assumptions (4)
  • domain assumption Expert-verified proper-chain labels are a sufficiently independent reference standard for action and safety judgments at surveillance distance.
    Invoked throughout Section 3 Level 3 and Section 5.1 to define the 84.6% human reference and to score models.
  • domain assumption The facility safety rule book (55 general + site-specific rules) correctly defines ground-truth safety violations for the evaluated clips.
    Safety labels and CRG depend on this external rule book (Appendix A.4).
  • ad hoc to paper Taxonomic distance weights used in DWA (0 / 0.33 / 0.60 / 0.70–1.00) appropriately reflect safety severity of confusions.
    Defined in Appendix C.2.6; not derived from injury statistics.
  • standard math Standard evaluation metrics (accuracy, macro-F1, ECE with 10 equal-width bins on self-reported confidence) are appropriate for comparing VLMs on this task.
    Used throughout Section 4–5 without modification beyond the new diagnostics.
invented entities (5)
  • SteelBench dataset and 2-layer annotation schema independent evidence
    purpose: Provide real industrial CCTV evaluation instances with dense per-worker multi-aspect labels.
    New public asset; independent evidence is the released HF dataset itself.
  • Provenance-aware three-level audit protocol (CLR, CR, anchoring bias, provenance-stratified accuracy) independent evidence
    purpose: Quantify how VLM pre-fills influence labels and downstream model scores.
    Methodological invention of the paper; falsifiable by re-running the audit on other model-assisted benchmarks.
  • CRG (Compositional Reasoning Gap)
    purpose: Isolate safety-reasoning failures from perception failures by conditioning on correct action prediction.
    New diagnostic defined in Eq. (1); useful but paper-specific.
  • nAUDC (Normalised Area Under Degradation Curve)
    purpose: Summarise robustness across six natural visibility conditions relative to best-condition accuracy.
    New robustness summary; independent of any single accuracy number.
  • DRS (Deployment Readiness Score)
    purpose: Binary checklist aggregating five capability thresholds into a single diagnostic.
    Paper-defined composite; not a regulatory standard.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SteelBench: Evaluating Vision-Language Models in Real-World Industrial Environments." pith.science (2026). https://pith.science/paper/AKYMRSXN

@misc{pith2026260705264,
  author       = {Pith},
  title        = {Pith review of: SteelBench: Evaluating Vision-Language Models in Real-World Industrial Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AKYMRSXN}},
  note         = {Machine review of arXiv:2607.05264}
}
read the original abstract

Existing video benchmarks evaluate action recognition on consumer videos, egocentric recordings, or simulated industrial environments. They do not test vision-language models under the visual and procedural conditions of real industrial CCTV, where workers appear as distant figures amid dust, steam, low light, glare, occlusion, and overlapping activities. We introduce STEELBENCH, a diagnostic benchmark for industrial surveillance that jointly evaluates per-worker activity recognition, safety-rule reasoning, and annotation provenance. SteelBench contains 1,345 densely annotated clips, curated from 149 hours of operational plant footage and 10,024 candidate clips using temporal deduplication, class balancing, and visibility-aware stratified sampling. Each clip includes dense per-worker action labels, PPE attributes, spatial context, and safety-rule annotations. Because model-assisted annotation can shape the labels later used for model evaluation, SteelBench includes a provenance-aware audit protocol. The protocol measures label influence, evaluates sensitivity to ground-truth provenance, and reports a human reference from expert-reviewed labels. Applying this audit, we find that unaudited VLM-sourced ground truth can inflate same-family model accuracy by up to 17 percentage points. Across nine VLMs from four architectural families, the best model reaches only 42.6% action accuracy, compared with an 84.6% human benchmark. Performance also fragments across recognition, robustness, calibration, and safety reasoning. Even when models predict the correct action, 37-58% of cases still yield incorrect safety judgments, and no model passes more than 2 of 5 diagnostic checks. The dataset is publicly available on Hugging Face.

Figures

Figures reproduced from arXiv: 2607.05264 by the authors.

Figure 1
Figure 1. An illustration of SteelBench data representing real industrial surveillance conditions, with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. SteelBench data distributions. (a) Action class distribution. The long tail reflects genuine [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Provenance-aware audit protocol for model-assisted benchmark construction. The three lev [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Safety recall versus false alarm rate across 9 [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 4
Figure 4. Figure 4: Per-class accuracy (%) across 25 action classes and 9 models. Action groups separated [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: Protocol sensitivity analysis. (a) Prompt variants on a 150-clip stratified subset. (b) Frame [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Direction analysis by annotation field (2,829 comparisons across 177 proper-chain clips). [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Per-annotator direction analysis on action classification. Blue segments are productive [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: accuracy (%) by visibility condition across 9 models. Steam and smoke (bottom rows) [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]
Figure 10
Figure 10. Figure 10: Per-site accuracy (%) across 12 plant zones and 9 models. Site difficulty is the dominant [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]
Figure 11
Figure 11. Figure 11: Accuracy by frame count (150 stratified clips). Gemma 4 improves gradually from 1 to 15 [PITH_FULL_IMAGE:figures/full_fig_p030_11.png]
Figure 12
Figure 12. Figure 12: Layer 2 accuracy by worker count (1–5) for top 5 models. The sharpest drop is from 1 to [PITH_FULL_IMAGE:figures/full_fig_p031_12.png]
Figure 13
Figure 13. Figure 13: Action accuracy on blind vs anchored GT across all 9 models. Every model scores [PITH_FULL_IMAGE:figures/full_fig_p032_13.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SafeSceneReason: A Multimodal Reasoning Benchmark Connecting Industrial Hazards with Accident Knowledge

    cs.AI 2026-08 conditional novelty 7.0 of 10

    A 123,695-question multimodal benchmark combining executable safety scene graphs and accident-report evidence graphs shows vision-language models still fail at compositional industrial-safety reasoning.

Reference graph

Works this paper leans on

39 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Quo vadis, action recognition? a new model and the kinetics dataset

    João Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. InCVPR, 2017

    2017

  2. [2]

    ActivityNet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. ActivityNet: A large-scale video benchmark for human activity understanding. InCVPR, 2015

    2015

  3. [3]

    A V A: A video dataset of spatio-temporally localized atomic visual actions

    Chunhui Gu, Chen Sun, David A Ross, Carl V ondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. A V A: A video dataset of spatio-temporally localized atomic visual actions. InCVPR, 2018

    2018

  4. [4]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman et al. Ego4d: Around the world in 3,000 hours of egocentric video. InCVPR, 2022

    2022

  5. [5]

    Assembly101: A large-scale multi-view video dataset for understanding procedural activities

    Fadime Sener, Dipika Chatterjee, Daniel Sheber, et al. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. InCVPR, 2022

    2022

  6. [6]

    IndustryEQA: Pushing the frontiers of embodied question answering in industrial scenarios

    Yifan Li, Yuhang Chen, Anh Dao, Lichi Li, Zhongyi Cai, Zhen Tan, Tianlong Chen, and Yu Kong. IndustryEQA: Pushing the frontiers of embodied question answering in industrial scenarios. InNeurIPS Datasets and Benchmarks, 2025

    2025

  7. [7]

    SH17: A dataset for human safety and personal protective equipment detection in manufacturing industry.Journal of Safety Science and Resilience, 2024

    Hafiz Mughees Ahmad et al. SH17: A dataset for human safety and personal protective equipment detection in manufacturing industry.Journal of Safety Science and Resilience, 2024

    2024

  8. [8]

    Koi Xiaowen Guo, Peter Kok-Yiu Wong, Jack C. P. Cheng, Chak-Fu Chan, Pak Him Leung, and Xingyu Tao. Enhancing visual-llm for construction site safety compliance via prompt engineering and bi-stage retrieval- augmented generation.Automation in Construction, 2025. URL https://api.semanticscholar.org/ CorpusID:281021165

    2025

Show all 39 references
  1. [9]

    Vision language model for interpretable and fine-grained detection of safety compliance in diverse workplaces.Expert Systems with Applications, 265:125769, 11 2024

    Zhiling Chen, Hanning Chen, Mohsen Imani, Ruimin Chen, and Farhad Imani. Vision language model for interpretable and fine-grained detection of safety compliance in diverse workplaces.Expert Systems with Applications, 265:125769, 11 2024. doi: 10.1016/j.eswa.2024.125769

    2024 doi

  2. [10]

    Human-LLM collabora- tive annotation through effective verification of LLM labels

    Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. Human-LLM collabora- tive annotation through effective verification of LLM labels. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. ACM, 2024. doi: 10.1145/3613904.3641960

    2024 doi

  3. [11]

    The state of data curation at NeurIPS: An assessment of dataset development practices in the datasets and benchmarks track

    Eshta Akhtar et al. The state of data curation at NeurIPS: An assessment of dataset development practices in the datasets and benchmarks track. InAdvances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2024. arXiv:2410.22473

    2024 arXiv

  4. [12]

    The ki- netics human action video dataset.ArXiv, abs/1705.06950, 2017

    Will Kay, João Carreira, Karen Simonyan, Brian Hu Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Apostol Natsev, Mustafa Suleyman, and Andrew Zisserman. The ki- netics human action video dataset.ArXiv, abs/1705.06950, 2017. URL https://...

    2017 arXiv

  5. [13]

    Ucf101: A dataset of 101 human actions classes from videos in the wild.ArXiv, abs/1212.0402, 2012

    Khurram Soomro, Amir Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild.ArXiv, abs/1212.0402, 2012. URL https://api.semanticscholar.org/ CorpusID:7197134

    2012 arXiv

  6. [14]

    Preference leakage: A contamination problem in LLM-as-a-judge

    Howie Wong et al. Preference leakage: A contamination problem in LLM-as-a-judge. InInternational Conference on Learning Representations (ICLR), 2026. arXiv:2502.01534

    2026

  7. [15]

    Just put a human in the loop? investigating LLM-assisted annotation for subjective tasks

    Nico Schroeder, Subhajit Roy, and Jad Kabbara. Just put a human in the loop? investigating LLM-assisted annotation for subjective tasks. InFindings of the Association for Computational Linguistics (ACL), 2025

    2025

  8. [16]

    Datasheets for datasets.Communications of the ACM, 64(12): 86–92, 2021

    Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets.Communications of the ACM, 64(12): 86–92, 2021

    2021

  9. [17]

    The MECCANO dataset: Understanding human-object interactions from egocentric videos in an industrial-like domain

    Francesco Ragusa, Antonino Furnari, Salvatore Livatino, and Giovanni Maria Farinella. The MECCANO dataset: Understanding human-object interactions from egocentric videos in an industrial-like domain. In WACV, 2021

    2021

  10. [18]

    Toyota smarthome: Real-world activities of daily living

    Srijan Das, Rui Dai, Michal Koperski, Luca Minciullo, Lorenzo Garattoni, Francois Bremond, and Gianpiero Francesca. Toyota smarthome: Real-world activities of daily living. InICCV, 2019. 11

    2019

  11. [19]

    Real-world anomaly detection in surveillance videos

    Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In CVPR, 2018

    2018

  12. [20]

    Future frame prediction for anomaly detection — a new baseline

    Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao. Future frame prediction for anomaly detection — a new baseline. InCVPR, 2018

    2018

  13. [21]

    iSafetyBench: A video-language benchmark for safety in industrial environments

    Raiyaan Abdullah, Yogesh Singh Rawat, and Shruti Vyas. iSafetyBench: A video-language benchmark for safety in industrial environments. InIEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2025

    2025

  14. [22]

    Inspecsafe-v1: A multimodal benchmark for safety assessment in industrial inspection scenarios, 2026

    Zeyi Liu, Shuang Liu, Jihai Min, Zhaoheng Zhang, Jun Cen, Pengyu Han, Songqiao Hu, Zihan Meng, Xiao He, and Donghua Zhou. Inspecsafe-v1: A multimodal benchmark for safety assessment in industrial inspection scenarios, 2026

    2026

  15. [23]

    MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI

    Xiang Yue et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. InCVPR, 2024

    2024

  16. [24]

    Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis

    Chaoyou Fu et al. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis. InCVPR, 2025

    2025

  17. [25]

    MVBench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li et al. MVBench: A comprehensive multi-modal video understanding benchmark. InCVPR, 2024

    2024

  18. [26]

    Benchmarking neural network robustness to common corruptions and perturbations

    Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. InICLR, 2019

    2019

  19. [27]

    NaturalBench: Evaluating vision-language models on natural adversarial samples

    Baiqi Li et al. NaturalBench: Evaluating vision-language models on natural adversarial samples. In NeurIPS Datasets and Benchmarks, 2024

    2024

  20. [28]

    Fields/clip

    Suryanarayana Reddy Yarrabothula, Vaibhav Kurrey, Mayank Nagar, and Gagan Raj Gupta. Industrial safety violation detection using action recognition. InProceedings of the 8th International Conference on Data Science and Management of Data (12th ACM IKDD CODS and 30th COMAD), 20...

    2025

  21. [29]

    Identify all visible workers with physical descriptions

  22. [30]

    Classify each worker’s primary action from the 25-class taxonomy

  23. [31]

    Provide a free-text description of what the worker is doing

  24. [32]

    Assess spatial context using a fixed tag vocabulary

  25. [33]

    Assess 5 PPE items per worker

  26. [34]

    Classify coordination (solo, coordinated, concurrent)

  27. [35]

    Flag any visible safety violations

  28. [36]

    Report action transitions across the 8 frames

  29. [37]

    Assess occlusion level and source

  30. [38]

    worn” only if clearly visible; use “cannot_determine

    Tag visibility conditions for the scene The model returns a structured JSON response matching a predefined schema. The full prompt text (approximately 2,500 tokens) is included in the code release. C.3.2 Model Configurations Table 17 lists the evaluated models with provider, A...

    2025

  31. [39]

    Experts and the Safety Officer are nominated by the plant to support the research

    Institutional review board (IRB) approvals or equivalent for research with human subjects Answer: [N/A] 39 Justification: Annotators are research team members performing annotation tasks, not experimental subjects. Experts and the Safety Officer are nominated by the plant to s...

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.