Pith Number

pith:2MD3PLBM

pith:2023:2MD3PLBMT6QKU3XL2GAJZJJ6B7

not attested not anchored not stored refs resolved

SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension

Bohao Li, Guangzhi Wang, Rui Wang, Ying Shan, Yixiao Ge, Yuying Ge

SEED-Bench supplies 19K human-verified multiple-choice questions to measure multimodal LLMs on image and video comprehension across 12 dimensions.

arxiv:2307.16125 v2 · 2023-07-30 · cs.CL · cs.CV

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{2MD3PLBMT6QKU3XL2GAJZJJ6B7}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

SEED-Bench consists of 19K multiple choice questions with accurate human annotations (x 6 larger than existing benchmarks), which spans 12 evaluation dimensions including the comprehension of both the image and video modality.

C2weakest assumption

That the pipeline of automatic question generation plus manual verification produces questions that genuinely measure generative comprehension rather than artifacts of the generation process itself.

C3one line summary

SEED-Bench is a new benchmark of 19K multiple-choice questions for evaluating generative comprehension in multimodal LLMs across 12 image and video dimensions.

References

42 extracted · 42 resolved · 16 Pith anchors

[1] Scaling Instruction-Finetuned Language Models 2022 · arXiv:2210.11416

[2] Gpt-4 technical report 2023

[3] Introducing chatgpt 2022

[4] FastChat. Vicuna. https://github.com/lm-sys/FastChat, 2023 2023

[5] LLaMA: Open and Efficient Foundation Language Models 2023 · arXiv:2302.13971

Formal links

2 machine-checked theorem links

Cited by

117 papers in Pith

TriViewBench: Controlled Complexity Scaling for Multi-View Structural Reasoning in MLLMs

Yuvion VL: A Multimodal Foundation Model for Adversarial Content and AI Safety

Mind the Heads: Topological Representation Alignment for Multimodal LLMs

HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoning

NEST: Narrative Event Structures in Time for Long Video Understanding

Receipt and verification

First computed	2026-07-05T06:36:55.686764Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

d307b7ac2c9fa0aa6eebd1809ca53e0fc7d15b6e9b301c0acabd365aab3b8676

Aliases

arxiv: 2307.16125 · arxiv_version: 2307.16125v2 · doi: 10.48550/arxiv.2307.16125 · pith_short_12: 2MD3PLBMT6QK · pith_short_16: 2MD3PLBMT6QKU3XL · pith_short_8: 2MD3PLBM

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/2MD3PLBMT6QKU3XL2GAJZJJ6B7 \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: d307b7ac2c9fa0aa6eebd1809ca53e0fc7d15b6e9b301c0acabd365aab3b8676

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "8b5fdc45f6d246ff05679d76140b94cc8ef1bd3b82a6eaf00c67fead7297a2a5",
    "cross_cats_sorted": [
      "cs.CV"
    ],
    "license": "http://creativecommons.org/licenses/by/4.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2023-07-30T04:25:16Z",
    "title_canon_sha256": "3cfa260a255e77b76cec5139aa572843eb9b952de93949abd296a45d044f1902"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2307.16125",
    "kind": "arxiv",
    "version": 2
  }
}