Pith Number

pith:AFBU3EZF

pith:2026:AFBU3EZFBDF4FLVS3NLWZXFYNQ

not attested not anchored not stored refs pending

HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs

Ahmed Cherif

Systematic benchmarking reveals NLI Verification as the most effective method for detecting hallucinations in LLMs at an AUROC of 0.88.

arxiv:2605.02443 v2 · 2026-05-04 · cs.CL

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{AFBU3EZFBDF4FLVS3NLWZXFYNQ}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

NLI Verification achieves the highest overall AUROC of 0.88, while RAV achieves the second-highest AUROC of 0.66. HalluScore achieves a Pearson correlation of r = 0.41 with human expert judgments.

C2weakest assumption

That the 72 configurations, chosen models, and three domains sufficiently represent the full range of hallucination behaviors in instruction-following LLMs and that human judgments provide a stable ground truth for validating the new metric.

C3one line summary

HalluScan benchmark tests hallucination detectors on LLMs, identifies NLI Verification as top performer with 0.88 AUROC, and introduces HalluScore (r=0.41 with humans) plus a routing method for 2x cost savings.

Formal links

3 machine-checked theorem links

Receipt and verification

First computed	2026-05-25T02:02:16.154597Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

01434d932508cbc2aeb2db576cdcb86c38372400133e99ec29c243388ce2d812

Aliases

arxiv: 2605.02443 · arxiv_version: 2605.02443v2 · doi: 10.48550/arxiv.2605.02443 · pith_short_12: AFBU3EZFBDF4 · pith_short_16: AFBU3EZFBDF4FLVS · pith_short_8: AFBU3EZF

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/AFBU3EZFBDF4FLVS3NLWZXFYNQ \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 01434d932508cbc2aeb2db576cdcb86c38372400133e99ec29c243388ce2d812

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "a8d4aa271dcdba19a8e074e11c31c6c2314cc65b4663f6828f11fb536ca80ae9",
    "cross_cats_sorted": [],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2026-05-04T10:43:27Z",
    "title_canon_sha256": "85d837d6891f48efbec948a0daf4484ec0c1b6bb65df91440679fdb993fb9319"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2605.02443",
    "kind": "arxiv",
    "version": 2
  }
}