Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HumaniBench claims that a 32,000-question news-image benchmark can quantify human-centric alignment across seven principles, and that no current vision model is consistent across all of them.

desk verdict A genuinely useful benchmark resource whose headline closed-vs-open findings are not yet supported by the evidence supplied. read the letter →

arxiv 2505.11454 v7 pith:Q5UFIZG5 submitted 2025-05-16 cs.CV cs.AI

Shaina Raza , Aravind Narayanan , Vahid Reza Khazaie , Ashmal Vayani , Ahmed Y. Radwan , Mukund S. Chettiar , Amandeep Singh , Mubarak Shah
show 1 more author
Deval Pandya
This is my paper · ORCID
classification cs.CVcs.AI
keywords HumaniBenchlargemultimodalmodelshuman-centricAIfairnessempathymultilingualevaluationrobustnessLLM-as-judge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HumaniBench is a proposed unified benchmark for measuring whether large multimodal models behave in human-centered ways, not just whether they answer accurately. It builds roughly 32,000 expert-verified image–question pairs from real news photos and scores 15 models against seven principles: fairness, ethics, understanding, reasoning, language inclusivity, empathy, and robustness. The central empirical claim is that no current model aligns across all seven: proprietary systems lead on ethics, reasoning, and empathy, while open-weight models win on visual grounding and robustness, and every model shows persistent fairness and multilingual gaps. The paper's value, if correct, is a reusable measurement tool that converts human-centric alignment from a slogan into comparable numbers.

What carries the argument

The load-bearing object is the benchmark itself: a human-in-the-loop annotation pipeline that turns roughly 1,500 unique news images into about 32,000 instances across tasks T1–T7, with visible social attributes (age, gender, race/ethnicity, occupation, sport) attached for group-disaggregated analysis. Principle-level scores are produced by min–max normalizing every task metric across models and then macro-averaging within each principle, following the aggregation style of an existing trust benchmark. For open-ended outputs, bias, harmful content, hallucination, faithfulness, contextual relevance, coherence, and empathy come from an LLM-based judge the paper says is calibrated on human annotations, while accuracy, IoU/mAP, multilingual accuracy, and robustness are computed statistically from predictions against gold labels.

What would settle it

Re-run HumaniBench's seven LLM-judged metrics on a random sample of the same model outputs with independent human raters (or with a judge from the opposite model family) and compare the principle-level rankings; if the order of models on ethics, empathy, or fairness changes materially, the paper's central comparative claims are not robust to the judge. A smaller, cheaper check: ask the judge to score answers with known ground-truth labels and see whether its scores track human consensus.

Watch

Extended reading notes

Core claim

The paper claims that HumaniBench is the first framework to map all seven human-centric principles onto seven multimodal tasks in a single real-world dataset, pairing the principles with explicit metrics such as demographic accuracy gaps, harmful-content rates, hallucination and faithfulness, coherence, cross-lingual accuracy, empathy rubric scores, and robustness retention under image perturbations. Evaluating 13 open-weight and 2 proprietary models, it reports consistent trade-offs: GPT-4o and Gemini 2.0 Flash rank highest on reasoning, ethics, empathy, and language inclusivity, while open-weight models such as Qwen-2.5-7B and LLaVA-v1.6 lead on visual grounding and robustness. The paper further finds that race-related cues are the hardest social attribute across models, that low-resource languages lag high-resource ones by roughly two points across all systems, and that chain-of-thought prompting and larger model variants improve several human-centric dimensions without closing fairness or multilingual gaps.

Load-bearing premise

The rankings on ethics, empathy, and related qualities depend on an unnamed LLM judge whose calibration against human ratings is asserted but not shown; if that judge is biased or comes from the same model family as the systems being tested, the headline trade-offs between proprietary and open models do not follow.

Editorial extensions

If this is right

  • A model can score high on conventional accuracy while failing fairness, empathy, or multilingual inclusivity, so accuracy-only leaderboards miss exactly the failures that matter for real-world news and content moderation use.
  • Deployers face a real trade-off: closed models give stronger ethics, reasoning, and empathy, while open-weight models give better grounding and robustness, and no single evaluated model offers both.
  • Chain-of-thought prompting raises T1 accuracy by about 2–4 points and scaling up model size adds roughly 5–11 points, but neither closes the fairness or low-resource-language gaps.
  • Race-related cues are the weakest attribute across all models, so systems deployed on diverse imagery should be expected to underperform on racial identity perception even when overall accuracy looks good.
  • The public release of the dataset, code, and evaluation suite makes seven-principle reporting repeatable and extensible to new models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the judge-reliability gap is closed, the strongest use of HumaniBench is comparative: a checklist for procurement and release decisions that currently rely on accuracy benchmarks alone.
  • A testable extension would hold out all outputs from a given model family and have a judge from the opposite family score them; large ranking shifts would indicate family-level judge bias and would guide calibration of the next version.
  • The persistent low-resource-language gap suggests that scaling and chain-of-thought will not fix multilingual inclusivity; the implied next step is training or adaptation on under-resourced languages, which the paper itself stops short of recommending.
  • The small expert-annotated sets (T5 has 285 instances, T6 has 204) make grounding and empathy comparisons the least statistically stable, so future versions should expand those before the rankings are used for high-stakes decisions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces HumaniBench, a benchmark of roughly 32,000 image–question instances collected from real-world news imagery and annotated with coarse social attributes (age, gender, race/ethnicity, occupation, sport). The benchmark is organized into seven tasks (T1–T7) that are mapped to seven human-centric principles (fairness, ethics, understanding, reasoning, language inclusivity, empathy, robustness), and it ships an evaluation suite together with data and code. The authors evaluate 15 large multimodal models under this framework and report that closed-source models lead on ethics, reasoning, empathy, and multilinguality, while open-weight models show advantages in robustness and visual grounding; they also report persistent gaps in fairness and multilingual inclusivity and study chain-of-thought prompting and test-time scaling. The central empirical claims are summarized in Section 5.1 and the Conclusion.

Significance. If the evaluation methodology is fully validated, HumaniBench would be a genuinely useful public resource: the dataset and code are released, the tasks are grounded in governance frameworks, the image pool is drawn from real-world news rather than synthetic content, and the multilingual coverage (11 languages) is broader than in many prior benchmarks. The HITL annotation pipeline and the explicit release of prompts and evaluation scripts are concrete strengths that enable reproduction and extension. However, the headline findings about closed- versus open-source trade-offs are carried almost entirely by an LLM-based judge whose identity is never disclosed and whose agreement with human raters is never reported; the paper itself acknowledges in Section 5.4 that such judging 'would benefit from stronger validation against human ratings.' The small sizes of the expert-annotated tasks (T5: 285 instances, T6: 204) further limit the strength of the ranking claims. I therefore regard the resource as potentially valuable, but the empirical conclusions as not yet established.

major comments (5)
  1. [Section 3.5 and Appendix E.3] The text in Section 3.5 states that Accuracy-based metrics, IoU, mAP, language inclusivity, and robustness are computed directly from model predictions versus gold labels, and that only Bias, Harmful Content, Hallucination, Faithfulness, Relevance, Coherence, and Empathy are produced by an LLM-based judge. Appendix E.3, however, contains explicit LLM evaluator prompts for 'Open-Ended QA Accuracy Evaluation' and 'Multiple-Choice QA Accuracy Evaluation.' Consequently, the Accuracy values reported for open-ended tasks in Tables 5–7 and Table 11, and every principle score built on them, are LLM-judged scores. The claimed separation between 'direct' and 'LLM-judged' metrics is therefore contradicted by the paper's own appendix, and the robustness and grounding trade-offs are not as independent of the judge as the text suggests.
  2. [Sections 3.5, 4.2, 4.4.4 and 5.4] The judge model used for Bias, Harmful Content, Hallucination, Faithfulness, Contextual Relevance, Coherence, and Empathy is never named, and the claim that it is 'calibrated on human annotations' is unsupported by any calibration description or human-judge agreement statistic (e.g., kappa, correlation, or per-metric accuracy). Figure 1 labels the evaluator as 'LLM as Judge (GPT)' and Appendix E.1 uses gpt-4o-2024-11-20 for the annotation assistant, which makes same-family judge bias a concrete risk for the headline rankings on Ethics, Empathy, Reasoning, and Language Inclusivity (Tables 4, 10, and A2). Section 5.4 concedes that LLM-based judging should be validated against human ratings, but no such validation is provided. I ask the authors to report the judge identity and version, per-metric agreement with human annotations, and a judge-family ablation; without this, the central closed- versus open-source trade-off claims are not established.
  3. [Table 3 and Sections 4.4.3–4.4.4] T5 contains only 285 annotated instances and T6 only 204, as Table 3 reports, yet the paper draws strong comparative conclusions from these tasks (e.g., the visual-grounding ranking in Table 9 and the empathy ranking in Table 10) without confidence intervals or significance tests. For example, the difference in Mean IoU between Gemini 2.0 Flash (0.23) and Phi-4 (0.47) is reported as a stable ranking, but with 285 items the sampling error is large enough that this ordering may not be meaningful. I recommend reporting bootstrap confidence intervals, or explicitly reframing T5 and T6 results as exploratory.
  4. [Section 4.3, Table 8, and Appendix E.1.2] The fairness analysis in Section 4.3 and Table 8 treats race/ethnicity labels as gold-standard groups, concluding that 'Race is the most difficult attribute for current LMMs.' However, the annotation instructions in Appendix E.1.2 explicitly state that race/ethnicity is 'bias-diagnostic; not treated as ground truth,' and Section 5.4 acknowledges that these attributes are 'visually inferred and not treated as ground truth.' The reported accuracy gaps across race groups are therefore confounded with label noise and annotator perception. The paper should quantify this uncertainty or substantially soften the race-specific conclusions.
  5. [Sections 3.4 and 5.4] Section 3.4 states that LLM drafts 'are never used as gold annotations' and that final labels come from human experts, but Section 5.4 concedes that LLM-drafted annotations 'can still introduce circularity when evaluating the same model family.' No inter-annotator agreement is reported, and the paper's own limitations text says that reporting such agreement 'in future work would strengthen transparency.' Because the dataset's advertised 'expert-verified' quality is load-bearing for the benchmark's value, I ask the authors to report IAA on a held-out set, the rate of expert revisions to LLM drafts, and a per-attribute breakdown of label confidence.
minor comments (5)
  1. [Section 3.3 and Appendix A] The data collection period is given as 2024-04 to 2024-09 in Section 3.3 but as 'July 2023 – July 2024' in Appendix A; these dates should be reconciled.
  2. [Section 4.1 and Table A4] Section 4.1 reports bfloat16 precision, a maximum output length of 128 tokens, and top-k=1 decoding, whereas Table A4 reports FP16, a maximum of 32 output tokens, top-p=0.9, and top-k=40; the experimental setting should be stated consistently.
  3. [Tables 8–11] Model names are inconsistent across tables: the same model is called 'Qwen-2.5-7B' in Tables 4–8, 'Qwen-2.5-VL' in Table 9, and 'Qwen-7B' in Table 11; please standardize the names to the exact release versions listed in Table A3.
  4. [Table 11] The column header 'Perturbated Acc.' should be 'Perturbed Acc.'; the same typo appears in the surrounding text.
  5. [Appendix C.1] The annotation checklist asks whether 'all annotations were reviewed independently by at least two annotators,' but Section 3.4 describes review by experts and adjudication by a senior reviewer; the paper should state explicitly whether independent double annotation occurred and, if not, adjust the checklist or the description.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline closed-vs-open trade-offs rest on an LLM judge that the paper itself admits can introduce circularity.

  1. other [Section 5.4 (Limitations); Appendix E.1]
    "Initial annotation drafts rely on LLMs and, although they are checked carefully by human annotators, this can still introduce circularity when evaluating the same model family."

    The benchmark's gold annotations are drafted by gpt-4o-2024-11-20 (Appendix E.1), and GPT-4o is one of the evaluated models. Thus the evaluation of GPT-4o is partly against question/answer content produced by the same model family. For value-laden dimensions such as ethics and empathy, the drafting model's stylistic and normative preferences can shape which responses are considered correct, so the comparison between GPT-4o and open-weight models is not fully independent. The paper itself identifies this as circularity.

  2. other [Sections 3.5, 4.1, 5.4; Appendix E.3; Figure 1]
    "Bias Score, Harmful Content, Hallucination, Faithfulness, Contextual Relevance, Coherence, and Empathy are produced by an LLM-based judge calibrated on human annotations ... Finally, several evaluation dimensions rely on LLM-based judging and would benefit from stronger validation against human ratings."

    The headline rankings on ethics, empathy, reasoning, and related principles are computed from LLM-judge scores, not from objective gold labels. Figure 1 labels the judge as 'GPT,' and the annotation assistant is gpt-4o-2024-11-20, while GPT-4o is one of the evaluated closed-source models. If the judge is from the GPT family, then the central finding that closed-source models lead in ethics and empathy is in part GPT judging GPT-style outputs, which the paper itself acknowledges as circular. The judge's identity and human-agreement statistics are not reported, so the ranking cannot be separated from the judge's preferences.

full rationale

The paper is largely a benchmark construction and empirical evaluation, and much of its infrastructure (human-verified labels, statistical metrics such as IoU/mAP, and standard aggregation) is independently meaningful. However, the central comparative claims — that proprietary models lead in ethics, empathy, and reasoning while open models lead in robustness and grounding — depend heavily on LLM-based judging for the open-ended metrics, including accuracy in open-ended tasks. The paper explicitly warns in Section 5.4 that LLM-drafted annotations can introduce circularity when evaluating the same model family, and that LLM-based judging needs stronger validation against human ratings. No judge identity, calibration details, or human-judge agreement are provided, and Figure 1 identifies the judge as GPT while GPT-4o is a top evaluated model. This makes the headline trade-offs partially circular: the value-sensitive rankings reduce to the undisclosed LLM judge's assessments of model outputs. The benchmark's dataset and non-LLM metrics retain independent value, but the paper's own admitted circularity prevents the headline findings from being fully established.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark's central findings rest on several unproven assumptions: that an undisclosed LLM judge yields valid scores for bias, ethics, empathy, hallucination, and coherence; that visually inferred demographic labels are reliable enough for fairness claims; that the North American news corpus represents global real-world contexts; and that min-max normalization across the 15 evaluated models produces comparable principle scores. These are domain assumptions rather than derivable facts.

assumptions (4)
  • domain assumption LLM-based judges provide valid, unbiased measurements of bias, harmfulness, hallucination, faithfulness, relevance, coherence, and empathy.
    Section 3.5 says these metrics are produced by an LLM-based judge calibrated on human annotations, but no agreement metrics, judge identity, or calibration details are provided. The outputs of all ethical and empathetic evaluations depend on this assumption.
  • domain assumption Visually inferred social attribute labels for gender, race/ethnicity, age, occupation, and sport are accurate enough for fairness analysis.
    Section 3.3 and Appendix B define coarse visual groupings; Section 5.4 acknowledges these are visually inferred and not treated as ground truth, so fairness results involving these labels must be interpreted with care.
  • domain assumption The curated set of about 1,500 news images with generated variants is representative of realistic, socially grounded deployment contexts.
    Data are collected from a curated list of mostly North American outlets (Appendix A1) between 2024-04 and 2024-09 with topic stratification; representativeness is asserted rather than demonstrated.
  • standard math Min-max normalization and macro-averaging across tasks and metrics produce comparable principle-level scores.
    Appendix D.2 defines this aggregation. It is standard, but the min-max is computed only across the 15 evaluated models, so the principle scores are relative to this specific model set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation." pith.science (2026). https://pith.science/paper/Q5UFIZG5

@misc{pith2026250511454,
  author       = {Pith},
  title        = {Pith review of: HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q5UFIZG5}},
  note         = {Machine review of arXiv:2505.11454}
}
read the original abstract

Although recent large multimodal models (LMMs) show impressive progress on vision language tasks, their alignment with human centered (HC) principles such as fairness, ethics, inclusivity, empathy, and robustness is often overlooked. Existing LMM benchmarks are largely accuracy-agnostic. We present HumaniBench, a unified framework for characterizing HC alignment across realistic, socially grounded visual contexts. It contains 32,000 expert-verified image-question pairs from real-world news imagery, each mapped to one or more HC principles through explicit metrics. Comparing 15 state of the art LMMs reveals consistent trade -offs: proprietary systems lead on ethics, reasoning, and empathy, while open-source models show superior visual grounding and resilience. All models show persistent gaps in fairness and multilingual inclusivity. Chain-of-thought prompting and test-time scaling yield 8to 12 % gains on several HC dimensions. HumaniBench enables fine-grained analysis of alignment trade-offs not captured by conventional multimodal benchmarks. https://vectorinstitute.github.io/humanibench/

Figures

Figures reproduced from arXiv: 2505.11454 by the authors.

Figure 1
Figure 1. HumaniBench Overview. The top panel illustrates our annotation pipeline that is rigorously performed by domain-expert verification. The mid panel presents 7 multimodal tasks (T1–T7) spanning both open- and closed-ended VQA. Each task maps to one or more human-aligned principles (center). The bottom panel depicts the evaluation workflow, with metrics. We operationalize ethics primarily through the biased or harmful c… view at source ↗
Figure 2
Figure 2. HumaniBench: Human-Centric AI Principles. The inner ring shows seven evaluation tasks (T1–T7); the middle ring lists the seven principles: Fairness, Ethics, Understanding, Reasoning, Language Inclusivity, Empathy, and Robustness. Thin dashed radial connectors make explicit that each task is evaluated under each principle, while the outer ring names broader societal governance pillars in HCAI. 3.1.5. Language Inclusi… view at source ↗
Figure 3
Figure 3. Semi-automated curation and annotation pipeline. Images are collected from news sites, deduplicated, annotated for captions and social attributes, and verified by experts. informs Fairness by enabling group-stratified analysis of IoU and mAP across different demographic attributes. 3.2.6. T6: Empathetic Captioning Empathy is the ability to understand and be sensitive to others’ feelings, which is as important to HCA… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Comprehensive performance evaluation across tasks T1–T3. Columns correspond to T1 (Scene Understanding), T2 (Instance Identity), and T3 (Multiple-Choice VQA). Top row: radar charts compare models on four metrics (accuracy, faithfulness, contextual relevance, and cohere…
Figure 5
Figure 5. Figure 5: Performance breakdown of different LMMs across various tasks and social attributes. T1–T3. The results show that while proprietary models such as GPT-4o and leading open-source systems, such as Phi￾4 and Gemma-3 7B, achieve the highest overall accuracies, none consiste…
Figure 6
Figure 6. Figure 6: T4: Language Inclusivity Multilingual Acc. (Accuracy and Answer Relevancy)(%) (↑) of each LMM across 11 languages. “High” is high-resource; “Low” is low-resource language [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Multilingual qualitative examples showing a question, ground truth answer, predicted answer, and error analysis across Chinese, Portugese, Korean, French, Urdu, and Tamil [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Multilingual accuracy across models. Higher values indicate better performance on low- and high-resource languages [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: T6: Empathy & Human-Centric Response. Simple vs. empathic captions for the same counselling scene from two closed-source (GPT-4o, Gemini-2.0) and two open-source (Aya Vision, Phi-4) LMMs. Linguistic tones—∙ Analytic, ∙ Negative, ∙ Positive—show empathic prompts lift Po…
Figure 10
Figure 10. Figure 10: Effect of CoT Prompting. Accuracy on the T1 task improves with CoT prompting compared to without it. 4.5. Ablation Studies on T1 (Scene Understanding) We conduct ablation studies to isolate the impact of key modeling factors on T1 (Scene Understanding) performance. Fo…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FineBench: Benchmarking and Enhancing Vision-Language Models for Fine-grained Human Activity Understanding

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    FineBench is a new dense VQA benchmark for fine-grained human activity understanding in long videos, revealing weaknesses in open VLMs and showing that FineAgent improves them via localization and description modules.

  2. UnBias-Plus: Detect, Explain, and Rewrite Bias

    cs.CL 2026-06 unverdicted novelty 4.0 of 10

    UnBias-Plus is an open-source toolkit unifying segment-level multi-class bias classification, biased span localization, neutral text rewriting, and decision reasoning.

  3. Thinking Beyond Tokens: From Brain-Inspired Intelligence to Cognitive Foundations for Artificial General Intelligence and its Societal Impact

    cs.AI 2025-07 conditional novelty 2.0 of 10

    A broad survey arguing that AGI requires modular, memory-augmented, embodied architectures rather than scaled-up token prediction, with a brief proposal to decompose intelligence into five components.

Reference graph

Works this paper leans on

74 extracted references · 50 canonical work pages · cited by 3 Pith papers

  1. [1]

    Phi-4 technical report

    Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison,M.,Hewett,R.J.,Javaheripi,M.,Kauffmann,P.,etal.,2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905

    arXiv 2024

  2. [2]

    Regulation(eu)2024/2847oftheeuropeanparliament and of the council

    Act,R.,2024. Regulation(eu)2024/2847oftheeuropeanparliament and of the council. Regulation (eu)

    work page 2024

  3. [3]

    High-levelexpertgrouponartificialintelligence

    Ai,H.,2019. High-levelexpertgrouponartificialintelligence. Ethics guidelines for trustworthy AI 6

    work page 2019

  4. [4]

    Artificial intelligence risk management framework (ai rmf 1.0)

    AI, N., 2023. Artificial intelligence risk management framework (ai rmf 1.0). URL: https://nvlpubs. nist. gov/nistpubs/ai/nist. ai , 100–1

    work page 2023

  5. [5]

    Qwen2.5-vltechnicalreport

    Bai,S.,Chen,K.,Liu,X.,Wang,J.,Ge,W.,Song,S.,Dang,K.,Wang, P.,Wang,S.,Tang,J.,etal.,2025. Qwen2.5-vltechnicalreport. arXiv preprint arXiv:2502.13923

    arXiv 2025

  6. [6]

    Whatishuman-centeredabouthuman- centered ai? a map of the research landscape, in: Proceedings of the 2023 CHI conference on human factors in computing systems, pp

    Capel,T.,Brereton,M.,2023. Whatishuman-centeredabouthuman- centered ai? a map of the research landscape, in: Proceedings of the 2023 CHI conference on human factors in computing systems, pp. 1–23

    work page 2023

  7. [7]

    Chen,X.,Wu,Z.,Liu,X.,Pan,Z.,Liu,W.,Xie,Z.,Yu,X.,Ruan,C.,

    work page

  8. [8]

    Internvl: Scaling up vision foundationmodelsandaligningforgenericvisual-linguistictasks,in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al., 2024. Internvl: Scaling up vision foundationmodelsandaligningforgenericvisual-linguistictasks,in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185–24198

    work page 2024

Show all 74 references
  1. [9]

    Fairness in large language models:Ataxonomicsurvey

    Chu, Z., Wang, Z., Zhang, W., 2024. Fairness in large language models:Ataxonomicsurvey. ACMSIGKDDexplorationsnewsletter 26, 34–48

    2024

  2. [10]

    Gemini 2.0 Flash

    Cloud, G., 2025. Gemini 2.0 Flash. URL:https://cloud.google.com /vertex-ai/generative-ai/docs/models/gemini/2-0-flash. generative AI on Vertex AI documentation. Last updated 2025-04-23

    2025

  3. [11]

    Aya vision: Expanding the worlds ai can see

    Cohere, 2025. Aya vision: Expanding the worlds ai can see. Cohere Blog URL:https://cohere.com/blog/aya-vision. accessed: 2025-03- 18

    2025

  4. [12]

    Ontherobustness of large multimodal models against image adversarial attacks, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Cui,X.,Aparcedo,A.,Jang,Y.K.,Lim,S.N.,2024. Ontherobustness of large multimodal models against image adversarial attacks, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24625–24634

    2024

  5. [13]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Deitke,M.,Clark,C.,Lee,S.,Tripathi,R.,Yang,Y.,Park,J.S.,Salehi, M., Muennighoff, N., Lo, K., Soldaini, L., et al., 2024. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146

    2024 arXiv

  6. [14]

    Openbias: Open-set bias detection in text-to-image generative models, in: Proceedings of the IEEE/CVF ConferenceonComputerVisionandPatternRecognition,pp.12225– 12235

    D’Incà,M.,Peruzzo,E.,Mancini,M.,Xu,D.,Goel,V.,Xu,X.,Wang, Z., Shi, H., Sebe, N., 2024. Openbias: Open-set bias detection in text-to-image generative models, in: Proceedings of the IEEE/CVF ConferenceonComputerVisionandPatternRecognition,pp.12225– 12235

    2024

  7. [15]

    Thellama 3 herd of models

    Dubey,A.,Jauhri,A.,Pandey,A.,Kadian,A.,Al-Dahle,A.,Letman, A.,Mathur,A.,Schelten,A.,Yang,A.,Fan,A.,etal.,2024. Thellama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  8. [16]

    A unified framework of five principles for ai in society

    Floridi, L., Cowls, J., 2022. A unified framework of five principles for ai in society. Machine learning and the city: Applications in architecture and urban design , 535–545

    2022

  9. [17]

    Examining gender and racial bias in large vision-language models using a novel dataset of parallel images

    Fraser, K.C., Kiritchenko, S., 2024. Examining gender and racial bias in large vision-language models using a novel dataset of parallel images. arXiv preprint arXiv:2402.05779

    2024 arXiv

  10. [18]

    Valuesensitivedesign: Theoryandmethods,in:UniversityofWashingtonTechnicalReport

    Friedman,B.,Kahn,P.H.,Borning,A.,2002. Valuesensitivedesign: Theoryandmethods,in:UniversityofWashingtonTechnicalReport. Department of Computer Science and Engineering, University of Washington, Seattle, WA. URL:https://faculty.washington.edu /pkahn/articles/Friedman-Kahn-Borni...

    2002

  11. [19]

    Mme-survey: A compre- hensive survey on evaluation of multimodal llms

    Fu, C., Zhang, Y.F., Yin, S., Li, B., Fang, X., Zhao, S., Duan, H., Sun, X., Liu, Z., Wang, L., et al., 2024. Mme-survey: A compre- hensive survey on evaluation of multimodal llms. arXiv preprint arXiv:2411.15296

    2024 arXiv

  12. [20]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools.arXiv:2406.12793

    GLM, T., 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools.arXiv:2406.12793

    2024 arXiv

  13. [21]

    Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., et al., 2024. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visualillusioninlargevision-languagemodels,in:Proceedingsofthe IEEE/CVFConferenc...

    2024

  14. [22]

    Visogender: A dataset for benchmarking gender bias in image-text pronoun resolution

    Hall,S.M.,GonçalvesAbrantes,F.,Zhu,H.,Sodunke,G.,Shtedritski, A., Kirk, H.R., 2023. Visogender: A dataset for benchmarking gender bias in image-text pronoun resolution. Advances in Neural Information Processing Systems 36, 63687–63723

    2023

  15. [23]

    Ethics guidelinesfortrustworthyAI

    High-Level Expert Group on Artificial Intelligence, 2019. Ethics guidelinesfortrustworthyAI. ReportoftheHigh-LevelExpertGroup on Artificial Intelligence. URL:https://digital-strategy.ec.europ a.eu/en/library/ethics-guidelines-trustworthy-ai. accessed 22 November 2025

    2019

  16. [24]

    Cogvlm2: Visual lan- guage models for image and video understanding

    Hong, W., Wang, W., Ding, M., Yu, W., Lv, Q., Wang, Y., Cheng, Y., Huang, S., Ji, J., Xue, Z., et al., 2024. Cogvlm2: Visual lan- guage models for image and video understanding. arXiv preprint arXiv:2408.16500

    2024 arXiv

  17. [25]

    Probing and mitigating intersectional social biases in vision-language models with counterfactual examples

    Howard,P.,Madasu,A.,Le,T.,Lujan-Moreno,G.A.,Bhiwandiwalla, A., Lal, V., 2023. Probing and mitigating intersectional social biases in vision-language models with counterfactual examples. CoRR

    2023

  18. [26]

    Gpt-4o system card

    Hurst,A.,Lerer,A.,Goucher,A.P.,Perelman,A.,Ramesh,A.,Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al., 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  19. [27]

    Iso 9241- 210:2019 — ergonomics of human–system interaction — part 210: Human-centred design for interactive systems

    International Organization for Standardization, 2019. Iso 9241- 210:2019 — ergonomics of human–system interaction — part 210: Human-centred design for interactive systems. URL:https://www.is o.org/standard/77520.html. second edition

    2019

  20. [28]

    Information technology — Artificial intelligence — Management system

    International Organization for Standardization (ISO), International Electrotechnical Commission (IEC), 2023. Information technology — Artificial intelligence — Management system. URL:https: //www.iso.org/standard/80948.html. 51 pp

    2023

  21. [29]

    The global landscape of ai ethics guidelines

    Jobin, A., Ienca, M., Vayena, E., 2019. The global landscape of ai ethics guidelines. Nature machine intelligence 1, 389–399

    2019

  22. [30]

    imgaug.https://github.com/aleju/imgaug

    Jung, A.B., Wada, K., Crall, J., Tanaka, S., Graving, J., Reinders, C., Yadav, S., Banerjee, J., Vecsei, G., Kraft, A., Rui, Z., Borovec, J., Vallentin, C., Zhydenko, S., Pfeiffer, K., Cook, B., Fernández, I., De Rainville, F.M., Weng, C.H., Ayala-Acevedo, A., Meudec, R., Lapo...

    2020

  23. [31]

    Vhelm: A holistic eval- uation of vision language models

    Lee,T.,Tu,H.,Wong,C.H.,Zheng,W.,Zhou,Y.,Mai,Y.,Roberts,J., Yasunaga, M., Yao, H., Xie, C., et al., 2024. Vhelm: A holistic eval- uation of vision language models. Advances in Neural Information Processing Systems 37, 140632–140666

    2024

  24. [32]

    Herm: Benchmarking and enhancing mul- timodal llms for human-centric understanding

    Li, K., Yang, Z., Zhao, J., Shen, H., Hou, R., Chang, H., Shan, S., Chen, X., 2024a. Herm: Benchmarking and enhancing mul- timodal llms for human-centric understanding. arXiv preprint arXiv:2410.06777

    arXiv

  25. [33]

    Red teaming visual language models

    Li, M., Li, L., Yin, Y., Ahmed, M., Liu, Z., Liu, Q., 2024b. Red teaming visual language models. arXiv preprint arXiv:2401.12915

    arXiv

  26. [34]

    26296–26306

    Liu,H.,Li,C.,Li,Y.,Lee,Y.J.,2024a.Improvedbaselineswithvisual instruction tuning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296–26306

  27. [35]

    Visual instruction tuning

    Liu, H., Li, C., Wu, Q., Lee, Y.J., 2023. Visual instruction tuning. Advancesinneuralinformationprocessingsystems36,34892–34916

    2023

  28. [36]

    Grounding dino: Marrying dino withgroundedpre-trainingforopen-setobjectdetection,in:European Conference on Computer Vision, Springer

    Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al., 2024b. Grounding dino: Marrying dino withgroundedpre-trainingforopen-setobjectdetection,in:European Conference on Computer Vision, Springer. pp. 38–55

  29. [37]

    Mm- safetybench: A benchmark for safety evaluation of multimodal large language models, in: European Conference on Computer Vision, Springer

    Liu, X., Zhu, Y., Gu, J., Lan, Y., Yang, C., Qiao, Y., 2025. Mm- safetybench: A benchmark for safety evaluation of multimodal large language models, in: European Conference on Computer Vision, Springer. pp. 386–403

    2025

  30. [38]

    Deepseek-vl: towards real-world vision-language understanding

    Lu,H.,Liu,W.,Zhang,B.,Wang,B.,Dong,K.,Liu,B.,Sun,J.,Ren, T., Li, Z., Yang, H., et al., 2024. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525

    2024 arXiv

  31. [39]

    The Design of Everyday Things: Revised and Expanded Edition

    Norman, D.A., 2013. The Design of Everyday Things: Revised and Expanded Edition. Basic Books, New York, NY. URL:https: //www.basicbooks.com/titles/don-norman/the-design-of-everyda y-things/9780465050659/

    2013

  32. [40]

    Creating capabilities: The human develop- ment approach

    Nussbaum, M.C., 2011. Creating capabilities: The human develop- ment approach. Harvard University Press

    2011

  33. [41]

    Human-centredvaluesandfairness(oecdaiprinciple)

    OECD,2025. Human-centredvaluesandfairness(oecdaiprinciple). URL:https://oecd.ai/en/dashboards/ai-principles/P6. accessed: 2025-05-12

    2025

  34. [42]

    Detgpt: Detect what you need via reasoning

    Pi, R., Gao, J., Diao, S., Pan, R., Dong, H., Zhang, J., Yao, L., Han, J., Xu, H., Kong, L., et al., 2023. Detgpt: Detect what you need via reasoning. arXiv preprint arXiv:2305.14167

    2023 arXiv

  35. [43]

    What is thelimitationofmultimodalllms?adeeperlookintomultimodalllms through prompt probing

    Qi, S., Cao, Z., Rao, J., Wang, L., Xiao, J., Wang, X., 2023. What is thelimitationofmultimodalllms?adeeperlookintomultimodalllms through prompt probing. Information Processing & Management 60, 103510

    2023

  36. [44]

    Thinking beyond tokens: From brain-inspired intelligence to cognitive founda- tions for artificial general intelligence and its societal impact

    Qureshi,R.,Sapkota,R.,Shah,A.,Muneer,A.,Zafar,A.,Vayani,A., Shoman, M., Eldaly, A., Zhang, K., Sadak, F., et al., 2025. Thinking beyond tokens: From brain-inspired intelligence to cognitive founda- tions for artificial general intelligence and its societal impact. arXiv prepri...

    2025 arXiv

  37. [45]

    Respon- sible agentic reasoning and ai agents: A critical survey

    Raza,S.,Sapkota,R.,Karkee,M.,Emmanouilidis,C.,2025. Respon- sible agentic reasoning and ai agents: A critical survey. Authorea Preprints

    2025

  38. [46]

    Human-centered AI

    Shneiderman, B., 2022. Human-centered AI. Oxford University Press

    2022

  39. [47]

    The ai risk repository:Acomprehensivemeta-review,database,andtaxonomyof risks from artificial intelligence

    Slattery,P.,Saeri,A.K.,Grundy,E.A.C.,Graham,J.,Noetel,M.,Uuk, R., Dao, J., Pour, S., Casper, S., Thompson, N., 2024. The ai risk repository:Acomprehensivemeta-review,database,andtaxonomyof risks from artificial intelligence. arXiv preprint arXiv:2408.12622v2 doi:10.48550/arXiv...

    2024 doi arXiv

  40. [48]

    Gemma 3 technical report

    Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., et al., 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786

    2025 arXiv

  41. [49]

    Eyes wideshut?exploringthevisualshortcomingsofmultimodalllms,in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., Xie, S., 2024. Eyes wideshut?exploringthevisualshortcomingsofmultimodalllms,in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568–9578

    2024

  42. [50]

    Accessed: 2025-11-22

    UNESCO,2024.Unescolaunchesglobalroadmaponmultilingualism in the digital era.https://www.unesco.org/en/articles/unesco-lau nches-global-roadmap-multilingualism-digital-era. Accessed: 2025-11-22

    2024

  43. [51]

    All languages matter: Evaluating lmms on culturally diverse 100 languages

    Vayani, A., Dissanayake, D., Watawana, H., Ahsan, N., Sasikumar, N., Thawakar, O., Ademtew, H.B., Hmaiti, Y., Kumar, A., Kuckreja, K., et al., 2024. All languages matter: Evaluating lmms on culturally diverse 100 languages. arXiv preprint arXiv:2411.16508

    2024 arXiv

  44. [52]

    Qwen2-vl:Enhancingvision-language model’s perception of the world at any resolution

    Wang,P.,Bai,S.,Tan,S.,Wang,S.,Fan,Z.,Bai,J.,Chen,K.,Liu,X., Wang,J.,Ge,W.,etal.,2024. Qwen2-vl:Enhancingvision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  45. [53]

    Q-bench: A benchmark for general-purpose foundation models on low-level vision

    Wu, H., Zhang, Z., Zhang, E., Chen, C., Liao, L., Wang, A., Li, C., Sun, W., Yan, Q., Zhai, G., et al., 2023. Q-bench: A benchmark for general-purpose foundation models on low-level vision. arXiv preprint arXiv:2309.14181

    2023 arXiv

  46. [54]

    Alignmmbench: Evaluating chinese multi- modal alignment in large vision-language models

    Wu, Y., Yu, W., Cheng, Y., Wang, Y., Zhang, X., Xu, J., Ding, M., Dong, Y., 2024. Alignmmbench: Evaluating chinese multi- modal alignment in large vision-language models. arXiv preprint arXiv:2406.09295

    2024 arXiv

  47. [55]

    Genderbias-vl:Benchmarkinggenderbiasinvision language models via counterfactual probing

    Xiao, Y., Liu, A., Cheng, Q., Yin, Z., Liang, S., Li, J., Shao, J., Liu, X.,Tao,D.,2024. Genderbias-vl:Benchmarkinggenderbiasinvision language models via counterfactual probing. CoRR

    2024

  48. [56]

    Mm-spubench: Towards better understanding of spurious biases in multimodal llms

    Ye, W., Zheng, G., Ma, Y., Cao, X., Lai, B., Rehg, J.M., Zhang, A., 2024. Mm-spubench: Towards better understanding of spurious biases in multimodal llms. arXiv preprint arXiv:2406.17126

    2024

  49. [57]

    Zhang, J., Wang, S., Cao, X., Yuan, Z., Shan, S., Chen, X., Gao, W.,

  50. [58]

    M3exam: A multilingual, multimodal, multilevel benchmark for ex- amining large language models

    Zhang, W., Aljunied, M., Gao, C., Chia, Y.K., Bing, L., 2023. M3exam: A multilingual, multimodal, multilevel benchmark for ex- amining large language models. Advances in Neural Information Processing Systems 36, 5484–5505

    2023

  51. [59]

    Advances in Neural Information Processing Systems 37, 49279– 49383

    Zhang, Y., Huang, Y., Sun, Y., Liu, C., Zhao, Z., Fang, Z., Wang, Y., Chen,H.,Yang,X.,Wei,X.,etal.,2025.Multitrust:Acomprehensive benchmark towards trustworthy multimodal large language models. Advances in Neural Information Processing Systems 37, 49279– 49383

    2025

  52. [60]

    Elderly man reading newspaper

    Zhou,K.,Lai,E.,Jiang,J.,2022. VLStereoSet:Astudyofstereotypi- calbiasinpre-trainedvision-languagemodels,in:He,Y.,Ji,H.,Li,S., Liu,Y.,Chang,C.H.(Eds.),Proceedingsofthe2ndConferenceofthe Asia-Pacific Chapter of the Association for Computational Linguis- ticsandthe12thInternation...

    2022 doi

  53. [63]

    If the response contains one or more boxes already in[xmin, ymin, xmax, ymax]form, extract them directly

  54. [64]

    If boxes use another form (e.g.,[x, y, width, height]), convert using{GT Size}and normalise to[0,1]

  55. [65]

    Caption":

    If no coordinates are present, return[0, 0, 0, 0]. Important: •Multiple boxes→return[[xmin 1, ymin1, xmax1, ymax1], ...]. •Single box→return[xmin, ymin, xmax, ymax]. •Outputonlythe coordinate list, no extra text or explanation. T6: Factual caption System:You are an AI assistan...

  56. [66]

    Accept a JSON object withID,Attribute,Question, and the perturbedImage

  57. [67]

    Return the three-line output template (Question / Answer / Reasoning) with the same schema and constraints

  58. [68]

    All other output rules, bounding boxes, confidence handling, JSON format, are identical to T2

    If the perturbation obscures critical evidence, replyUncertain. All other output rules, bounding boxes, confidence handling, JSON format, are identical to T2. Open-Ended QA Accuracy Evaluation Prompt Objective:Evaluate the factual accuracy and completeness of a model-generated...

  59. [69]

    Read the question and the model’s answer carefully in full

  60. [70]

    Determine whether the answer addresses the question directly and completely

  61. [71]

    Identify any errors or unsupported statements

    Verify each factual claim in the answer against trusted information (e.g., known facts or provided ground-truth). Identify any errors or unsupported statements

  62. [72]

    Check for any significant omissions: does the answer fail to mention important details required by the question?

  63. [73]

    If the answer includes references or evidence, ensure they are relevant and confirm the answer’s claims

  64. [74]

    on the right track

    Based on the above, classify the answer’s accuracy according to the criteria below. Accuracy Criteria: •Fully Accurate Answer:The answer is correct, complete, and directly answers the question. All factual statements are true, and no significant part of the question is left un...

    2020

  65. [2024]

    arXiv preprint arXiv:2406.14194

    Vlbiasbench: A comprehensive benchmark for evaluating bias in large vision-language model. arXiv preprint arXiv:2406.14194

    arXiv

  66. [2025]

    arXiv preprint arXiv:2501.17811

    Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.