{"id":"103cc10a-b4f9-4c78-b722-6921629209ec","arxiv_id":"2505.11454","paper_version":7,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"HumaniBench, a news-image benchmark with 32,000 expert-verified items across seven tasks, reveals that no evaluated multimodal model aligns well across all seven human-centric principles.","lead":"This paper introduces HumaniBench, a benchmark of 32,000 image-question pairs for evaluating how well vision-language models align with human-centered principles such as fairness, ethics, empathy, inclusivity, and robustness. It evaluates 15 models and finds trade-offs: proprietary systems lead on ethics and empathy, while open-source models show stronger visual grounding and robustness.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline trade-offs rest on an undisclosed, unvalidated LLM judge; the paper's own Section 5.4 admits circularity, so the closed-vs-open findings are not yet established.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the LLM-based judge is unnamed, unvalidated, and potentially circular. I agree. The paper's headline findings—closed-source leadership on ethics, empathy, and reasoning, and open-source advantages on robustness and grounding—are the central claims, and almost all of the underlying metrics flow through the LLM judge. The paper's own limitation section admits the circularity and the lack of human validation, so this is not an external objection to consensus but an internal evidential gap. The dataset and evaluation suite may still be useful, and the concern is fixable by publishing judge identity, calibration data, and human agreement, or by re-running a subset with human raters. No formal verification exists, and the released code cannot compensate for the hidden judge. Secondary weaknesses—tiny expert-annotated sets for T5/T6 (285 and 204 instances), no inter-annotator agreement, and the unsupported abstract claim of 8-12% gains from CoT/test-time scaling—do not displace the judge concern as the most load-bearing. Because the concern is real but repairable, the conditional verdict stands.","tokens_in":40486,"tokens_out":4406,"duration_ms":48095,"concrete_test":"Run a judge-swap plus human-rater validation on a stratified sample. Take 200 model outputs per model from T1, T2, T6, and T7 (covering GPT-4o, Gemini, Qwen, Phi, LLaVA, and a couple of mid-tier open models). Score the same outputs with the current undisclosed judge, an independent open-weight judge (e.g., Llama-3.3-70B), and 3-5 human raters using the Appendix E rubrics. Report rank correlations and pairwise agreement. If human-rater rankings of Ethics/Empathy do not reproduce a GPT-4o-first ordering, or if the open-weight judge ranks GPT-4o lower on those dimensions, the family-bias concern lands and the headline trade-offs must be revised. As part of the same check, require the authors to name the judge model and release the human-calibration data and agreement statistics referenced in Section 3.5.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that closed-source models lead in ethics, reasoning, and empathy while open-source models lead in robustness and grounding—is produced almost entirely by an LLM-based judge. Section 3.5 states that Bias Score, Harmful Content, Hallucination, Faithfulness, Contextual Relevance, Coherence, and Empathy are 'produced by an LLM-based judge calibrated on human annotations,' but the judge model is never named, the calibration protocol is not described, and no human-judge agreement statistic (kappa, correlation, or accuracy) is reported anywhere. Figure 1's 'LLM as Judge (GPT)' label and Appendix E.1's use of gpt-4o-2024-11-20 for the annotation assistant strongly suggest the judge is GPT-4o or a same-family model. If so, the top-line result is vulnerable to same-family preference: a GPT-4o judge may score GPT-4o outputs more favorably than equally good open-model outputs on subjective dimensions like empathy and ethics. The paper itself acknowledges in Section 5.4 that LLM-drafted annotations can introduce circularity when evaluating the same model family and that LLM-based judging 'would benefit from stronger validation against human ratings'; no such validation is supplied. Moreover, the open-ended Accuracy metrics used for T1/T2 and for T7 robustness are also produced by an LLM evaluator prompt (Appendix E.3), so even the 'accuracy-based' grounding and robustness trade-offs and the fairness gaps are affected by judge bias. Without judge identity, calibration details, and human-agreement evidence, the headline findings about cross-model alignment trade-offs are not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HumaniBench, a benchmark of roughly 32,000 image–question instances collected from real-world news imagery and annotated with coarse social attributes (age, gender, race/ethnicity, occupation, sport). The benchmark is organized into seven tasks (T1–T7) that are mapped to seven human-centric principles (fairness, ethics, understanding, reasoning, language inclusivity, empathy, robustness), and it ships an evaluation suite together with data and code. The authors evaluate 15 large multimodal models under this framework and report that closed-source models lead on ethics, reasoning, empathy, and multilinguality, while open-weight models show advantages in robustness and visual grounding; they also report persistent gaps in fairness and multilingual inclusivity and study chain-of-thought prompting and test-time scaling. The central empirical claims are summarized in Section 5.1 and the Conclusion.","tokens_in":40780,"tokens_out":5109,"duration_ms":49424,"significance":"If the evaluation methodology is fully validated, HumaniBench would be a genuinely useful public resource: the dataset and code are released, the tasks are grounded in governance frameworks, the image pool is drawn from real-world news rather than synthetic content, and the multilingual coverage (11 languages) is broader than in many prior benchmarks. The HITL annotation pipeline and the explicit release of prompts and evaluation scripts are concrete strengths that enable reproduction and extension. However, the headline findings about closed- versus open-source trade-offs are carried almost entirely by an LLM-based judge whose identity is never disclosed and whose agreement with human raters is never reported; the paper itself acknowledges in Section 5.4 that such judging 'would benefit from stronger validation against human ratings.' The small sizes of the expert-annotated tasks (T5: 285 instances, T6: 204) further limit the strength of the ranking claims. I therefore regard the resource as potentially valuable, but the empirical conclusions as not yet established.","major_comments":[{"comment":"The text in Section 3.5 states that Accuracy-based metrics, IoU, mAP, language inclusivity, and robustness are computed directly from model predictions versus gold labels, and that only Bias, Harmful Content, Hallucination, Faithfulness, Relevance, Coherence, and Empathy are produced by an LLM-based judge. Appendix E.3, however, contains explicit LLM evaluator prompts for 'Open-Ended QA Accuracy Evaluation' and 'Multiple-Choice QA Accuracy Evaluation.' Consequently, the Accuracy values reported for open-ended tasks in Tables 5–7 and Table 11, and every principle score built on them, are LLM-judged scores. The claimed separation between 'direct' and 'LLM-judged' metrics is therefore contradicted by the paper's own appendix, and the robustness and grounding trade-offs are not as independent of the judge as the text suggests.","section":"Section 3.5 and Appendix E.3"},{"comment":"The judge model used for Bias, Harmful Content, Hallucination, Faithfulness, Contextual Relevance, Coherence, and Empathy is never named, and the claim that it is 'calibrated on human annotations' is unsupported by any calibration description or human-judge agreement statistic (e.g., kappa, correlation, or per-metric accuracy). Figure 1 labels the evaluator as 'LLM as Judge (GPT)' and Appendix E.1 uses gpt-4o-2024-11-20 for the annotation assistant, which makes same-family judge bias a concrete risk for the headline rankings on Ethics, Empathy, Reasoning, and Language Inclusivity (Tables 4, 10, and A2). Section 5.4 concedes that LLM-based judging should be validated against human ratings, but no such validation is provided. I ask the authors to report the judge identity and version, per-metric agreement with human annotations, and a judge-family ablation; without this, the central closed- versus open-source trade-off claims are not established.","section":"Sections 3.5, 4.2, 4.4.4 and 5.4"},{"comment":"T5 contains only 285 annotated instances and T6 only 204, as Table 3 reports, yet the paper draws strong comparative conclusions from these tasks (e.g., the visual-grounding ranking in Table 9 and the empathy ranking in Table 10) without confidence intervals or significance tests. For example, the difference in Mean IoU between Gemini 2.0 Flash (0.23) and Phi-4 (0.47) is reported as a stable ranking, but with 285 items the sampling error is large enough that this ordering may not be meaningful. I recommend reporting bootstrap confidence intervals, or explicitly reframing T5 and T6 results as exploratory.","section":"Table 3 and Sections 4.4.3–4.4.4"},{"comment":"The fairness analysis in Section 4.3 and Table 8 treats race/ethnicity labels as gold-standard groups, concluding that 'Race is the most difficult attribute for current LMMs.' However, the annotation instructions in Appendix E.1.2 explicitly state that race/ethnicity is 'bias-diagnostic; not treated as ground truth,' and Section 5.4 acknowledges that these attributes are 'visually inferred and not treated as ground truth.' The reported accuracy gaps across race groups are therefore confounded with label noise and annotator perception. The paper should quantify this uncertainty or substantially soften the race-specific conclusions.","section":"Section 4.3, Table 8, and Appendix E.1.2"},{"comment":"Section 3.4 states that LLM drafts 'are never used as gold annotations' and that final labels come from human experts, but Section 5.4 concedes that LLM-drafted annotations 'can still introduce circularity when evaluating the same model family.' No inter-annotator agreement is reported, and the paper's own limitations text says that reporting such agreement 'in future work would strengthen transparency.' Because the dataset's advertised 'expert-verified' quality is load-bearing for the benchmark's value, I ask the authors to report IAA on a held-out set, the rate of expert revisions to LLM drafts, and a per-attribute breakdown of label confidence.","section":"Sections 3.4 and 5.4"}],"minor_comments":[{"comment":"The data collection period is given as 2024-04 to 2024-09 in Section 3.3 but as 'July 2023 – July 2024' in Appendix A; these dates should be reconciled.","section":"Section 3.3 and Appendix A"},{"comment":"Section 4.1 reports bfloat16 precision, a maximum output length of 128 tokens, and top-k=1 decoding, whereas Table A4 reports FP16, a maximum of 32 output tokens, top-p=0.9, and top-k=40; the experimental setting should be stated consistently.","section":"Section 4.1 and Table A4"},{"comment":"Model names are inconsistent across tables: the same model is called 'Qwen-2.5-7B' in Tables 4–8, 'Qwen-2.5-VL' in Table 9, and 'Qwen-7B' in Table 11; please standardize the names to the exact release versions listed in Table A3.","section":"Tables 8–11"},{"comment":"The column header 'Perturbated Acc.' should be 'Perturbed Acc.'; the same typo appears in the surrounding text.","section":"Table 11"},{"comment":"The annotation checklist asks whether 'all annotations were reviewed independently by at least two annotators,' but Section 3.4 describes review by experts and adjudication by a senior reviewer; the paper should state explicitly whether independent double annotation occurred and, if not, adjust the checklist or the description.","section":"Appendix C.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is best viewed as a benchmark-and-dataset contribution, and that part is potentially publishable. The empirical ranking section, however, cannot carry the paper's advertised conclusions until the LLM judge is identified and validated against human ratings, and until the small T5/T6 samples are handled with appropriate uncertainty quantification. If the authors cannot supply that validation, I would suggest reframing the empirical results as exploratory and moving the emphasis to the dataset and evaluation infrastructure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful resource, but the paper's central empirical claim—closed models lead in ethics/empathy, open models in grounding/robustness—is carried by an LLM judge whose identity, calibration, and human agreement are never reported. The paper admits as much in Section 5.4. So treat the headline as provisional until the judge is pinned down.\n\nWhat's genuinely new: the 32k news-image QA set with five social attribute labels, an 11-language split, seven tasks mapped to seven principles, and released data and code. The curation pipeline is described in enough detail to reproduce. The comparison table in Table 1 is a fair mapping of prior benchmarks. The limitations section is the most candid part of the paper—it flags circularity, small T5/T6 sets, visually inferred race labels, and the need for human validation. That honesty earns credit.\n\nWhere it's soft, in proportion:\n1. The LLM judge. Bias, hallucination, faithfulness, coherence, relevance, empathy, and even open-ended accuracy are scored by an LLM-based evaluator. The judge is never named. The 'calibrated on human annotations' claim has no calibration details or agreement statistics. Since GPT-4o is both a top-scoring model and the likely judge (Figure 1, Appendix E), same-family bias is a real risk. This is the load-bearing flaw for the ranking conclusions.\n2. The abstract's '8–12% gains' figure. The CoT ablations show +2–4%, and the scaling table shows +5.4 to +11.1. The number doesn't match either set cleanly.\n3. Small expert-verified tasks: T5 has 285 instances, T6 204. Fine for a pilot, thin for mAP/robustness conclusions. No confidence intervals anywhere.\n4. The 'technically new' issue is acknowledged in the paper itself: future versions should say what is new beyond the framing.\n\nDoes the central argument hold? The resource itself works as a benchmark—the data and code are real and reusable. The empirical trade-off story is plausible but not established.\n\nWho it's for: anyone building human-centric LMM evaluations; a useful complement to V-HELM and MultiTrust.\n\nRecommendation: deserves a serious referee. With judge disclosure, human-agreement numbers, corrected abstract, and softened claims, it's a solid contribution. I'd take a desk-reject off the table.","headline":"A genuinely useful benchmark resource whose headline closed-vs-open findings are not yet supported by the evidence supplied.","tokens_in":41356,"tokens_out":1870,"would_cite":true,"duration_ms":18734,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HumaniBench claims that a 32,000-question news-image benchmark can quantify human-centric alignment across seven principles, and that no current vision model is consistent across all of them.","keywords":["HumaniBench","large multimodal models","human-centric AI","fairness","empathy","multilingual evaluation","robustness","LLM-as-judge"],"falsifier":"Re-run HumaniBench's seven LLM-judged metrics on a random sample of the same model outputs with independent human raters (or with a judge from the opposite model family) and compare the principle-level rankings; if the order of models on ethics, empathy, or fairness changes materially, the paper's central comparative claims are not robust to the judge. A smaller, cheaper check: ask the judge to score answers with known ground-truth labels and see whether its scores track human consensus.","tokens_in":40311,"feed_emoji":"⚖️","tokens_out":6122,"duration_ms":58754,"temperature":0.7,"pith_summary":"HumaniBench is a proposed unified benchmark for measuring whether large multimodal models behave in human-centered ways, not just whether they answer accurately. It builds roughly 32,000 expert-verified image–question pairs from real news photos and scores 15 models against seven principles: fairness, ethics, understanding, reasoning, language inclusivity, empathy, and robustness. The central empirical claim is that no current model aligns across all seven: proprietary systems lead on ethics, reasoning, and empathy, while open-weight models win on visual grounding and robustness, and every model shows persistent fairness and multilingual gaps. The paper's value, if correct, is a reusable measurement tool that converts human-centric alignment from a slogan into comparable numbers.","feed_headline":"No vision model masters all human values, 15-model study finds","feed_subtitle":"Closed models lead on ethics and empathy; open models win on grounding. Fairness gaps persist for all 15.","key_machinery":"The load-bearing object is the benchmark itself: a human-in-the-loop annotation pipeline that turns roughly 1,500 unique news images into about 32,000 instances across tasks T1–T7, with visible social attributes (age, gender, race/ethnicity, occupation, sport) attached for group-disaggregated analysis. Principle-level scores are produced by min–max normalizing every task metric across models and then macro-averaging within each principle, following the aggregation style of an existing trust benchmark. For open-ended outputs, bias, harmful content, hallucination, faithfulness, contextual relevance, coherence, and empathy come from an LLM-based judge the paper says is calibrated on human annotations, while accuracy, IoU/mAP, multilingual accuracy, and robustness are computed statistically from predictions against gold labels.","core_discovery":"The paper claims that HumaniBench is the first framework to map all seven human-centric principles onto seven multimodal tasks in a single real-world dataset, pairing the principles with explicit metrics such as demographic accuracy gaps, harmful-content rates, hallucination and faithfulness, coherence, cross-lingual accuracy, empathy rubric scores, and robustness retention under image perturbations. Evaluating 13 open-weight and 2 proprietary models, it reports consistent trade-offs: GPT-4o and Gemini 2.0 Flash rank highest on reasoning, ethics, empathy, and language inclusivity, while open-weight models such as Qwen-2.5-7B and LLaVA-v1.6 lead on visual grounding and robustness. The paper further finds that race-related cues are the hardest social attribute across models, that low-resource languages lag high-resource ones by roughly two points across all systems, and that chain-of-thought prompting and larger model variants improve several human-centric dimensions without closing fairness or multilingual gaps.","pith_inferences":["If the judge-reliability gap is closed, the strongest use of HumaniBench is comparative: a checklist for procurement and release decisions that currently rely on accuracy benchmarks alone.","A testable extension would hold out all outputs from a given model family and have a judge from the opposite family score them; large ranking shifts would indicate family-level judge bias and would guide calibration of the next version.","The persistent low-resource-language gap suggests that scaling and chain-of-thought will not fix multilingual inclusivity; the implied next step is training or adaptation on under-resourced languages, which the paper itself stops short of recommending.","The small expert-annotated sets (T5 has 285 instances, T6 has 204) make grounding and empathy comparisons the least statistically stable, so future versions should expand those before the rankings are used for high-stakes decisions."],"forward_implications":["A model can score high on conventional accuracy while failing fairness, empathy, or multilingual inclusivity, so accuracy-only leaderboards miss exactly the failures that matter for real-world news and content moderation use.","Deployers face a real trade-off: closed models give stronger ethics, reasoning, and empathy, while open-weight models give better grounding and robustness, and no single evaluated model offers both.","Chain-of-thought prompting raises T1 accuracy by about 2–4 points and scaling up model size adds roughly 5–11 points, but neither closes the fairness or low-resource-language gaps.","Race-related cues are the weakest attribute across all models, so systems deployed on diverse imagery should be expected to underperform on racial identity perception even when overall accuracy looks good.","The public release of the dataset, code, and evaluation suite makes seven-principle reporting repeatable and extensible to new models."],"supporting_citations":[{"why":"Supplies the min–max normalization and macro-averaging procedure that turns heterogeneous task metrics into comparable principle-level scores.","marker":"[59]"},{"why":"Provides the perturbation types (blur, noise, compression, salt-and-pepper) used in the T7 robustness retention scores.","marker":"[30]"},{"why":"Proposes candidate bounding boxes for T5 visual grounding that human annotators accept or reject as gold references.","marker":"[36]"},{"why":"Cited as precedent for LLM-assisted annotation and for the hallucination metric's focus on language-driven visual illusions.","marker":"[21]"},{"why":"Cited with [21] as the workflow precedent for LLM drafting followed by human verification of gold labels.","marker":"[42]"},{"why":"The closest holistic benchmark whose multi-axis coverage HumaniBench positions itself against.","marker":"[31]"},{"why":"One of the governance frameworks from which the seven human-centric principles are consolidated.","marker":"[4]"},{"why":"System card for the top-ranked proprietary model whose scores anchor the closed-source comparison.","marker":"[26]"},{"why":"Documents the second proprietary model used as an upper-bound comparison in the evaluation.","marker":"[10]"}],"fun_headline_variants":["HumaniBench: 15 LMMs tested for fairness, ethics, empathy","Study: No AI vision model fully aligned with human values","LMMs trade off empathy vs grounding, HumaniBench shows","Fairness gap persists across 15 multimodal models","New benchmark reveals human-centric weaknesses in LMMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings on ethics, empathy, and related qualities depend on an unnamed LLM judge whose calibration against human ratings is asserted but not shown; if that judge is biased or comes from the same model family as the systems being tested, the headline trade-offs between proprietary and open models do not follow.","fun_headline_variants_meta":{"raw":{"variants":["HumaniBench: 15 LMMs tested for fairness, ethics, empathy","Study: No AI vision model fully aligned with human values","LMMs trade off empathy vs grounding, HumaniBench shows","Fairness gap persists across 15 multimodal models","New benchmark reveals human-centric weaknesses in LMMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000598,"raw_usage":{"total_tokens":2785,"prompt_tokens":920,"completion_tokens":1865,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":1780}},"tokens_in":536,"tokens_out":1865,"duration_ms":11271,"temperature":1.0,"reasoning_tokens":1780,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:52:31.531210+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run HumaniBench's seven LLM-judged metrics on a random sample of the same model outputs with independent human raters (or with a judge from the opposite model family) and compare the principle-level rankings; if the order of models on ethics, empathy, or fairness changes materially, the paper's central comparative claims are not robust to the judge. A smaller, cheaper check: ask the judge to score answers with known ground-truth labels and see whether its scores track human consensus.","supporting_citations":[{"cited_title":"imgaug.https://github.com/aleju/imgaug","cited_arxiv_id":null,"evidence_quote":"Provides the perturbation types (blur, noise, compression, salt-and-pepper) used in the T7 robustness retention scores."},{"cited_title":"Advances in Neural Information Processing Systems 37, 49279– 49383","cited_arxiv_id":null,"evidence_quote":"Supplies the min–max normalization and macro-averaging procedure that turns heterogeneous task metrics into comparable principle-level scores."},{"cited_title":"Grounding dino: Marrying dino withgroundedpre-trainingforopen-setobjectdetection,in:European Conference on Computer Vision, Springer","cited_arxiv_id":null,"evidence_quote":"Proposes candidate bounding boxes for T5 visual grounding that human annotators accept or reject as gold references."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited as precedent for LLM-assisted annotation and for the hallucination metric's focus on language-driven visual illusions."},{"cited_title":"Vhelm: A holistic eval- uation of vision language models","cited_arxiv_id":null,"evidence_quote":"The closest holistic benchmark whose multi-axis coverage HumaniBench positions itself against."},{"cited_title":"Artificial intelligence risk management framework (ai rmf 1.0)","cited_arxiv_id":null,"evidence_quote":"One of the governance frameworks from which the seven human-centric principles are consolidated."},{"cited_title":"Gemini 2.0 Flash","cited_arxiv_id":null,"evidence_quote":"Documents the second proprietary model used as an upper-bound comparison in the evaluation."}],"review_version":1}