Pith Number

pith:IPZIX56F

pith:2026:IPZIX56FVQVL2OXHZHMS2OHRNT

not attested not anchored not stored refs resolved

Creativity Bias: How Machine Evaluation Struggles with Creativity in Literary Translations

Ana Guerberof Arenas, Kyo Gerrits, Rik van Noord

Automatic evaluation metrics and LLM judges correlate poorly with professional translators on creativity in literary texts and bias toward machine outputs.

arxiv:2605.13596 v1 · 2026-05-13 · cs.CL

Open paper page JSON Open Graph Bundle Merged state What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{IPZIX56FVQVL2OXHZHMS2OHRNT}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

both AEMs and LLM-as-a-judge evaluations correlate poorly with professional evaluations on creativity, with LLM-as-a-judge showing a systematic bias in favour of machine-translated texts and penalising creative and culturally appropriate solutions.

C2weakest assumption

That detailed annotations by experienced professional literary translators constitute an objective and reliable ground truth for measuring creativity and translation quality across genres and modalities.

C3one line summary

Automatic evaluation tools for literary translations correlate poorly with expert human judgments on creativity and exhibit bias favoring machine-translated texts.

References

130 extracted · 130 resolved · 0 Pith anchors

[1] and Ullman, Jeffrey D 1972

[2] Interspeech 2006 --- Ninth International Conference on Spoken Language Processing , address= 2006

[3] Publications. 1983 , publisher= 1983

[4] Chandra and Dexter C 1981

[5] Coling 2008, 22nd International Conference on Computational Linguistics , address= 2008

Receipt and verification

First computed	2026-05-18T02:44:23.010234Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

43f28bf7c5ac2abd3ae7c9d92d38f16cc78ab49d2e0a413c186acacf0aa539b1

Aliases

arxiv: 2605.13596 · arxiv_version: 2605.13596v1 · doi: 10.48550/arxiv.2605.13596 · pith_short_12: IPZIX56FVQVL · pith_short_16: IPZIX56FVQVL2OXH · pith_short_8: IPZIX56F

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/IPZIX56FVQVL2OXHZHMS2OHRNT \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 43f28bf7c5ac2abd3ae7c9d92d38f16cc78ab49d2e0a413c186acacf0aa539b1

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "118075b308bfe2d85f63508597282bc4489f77d3f7d7cc990bd032ea4eb9ba70",
    "cross_cats_sorted": [],
    "license": "http://creativecommons.org/licenses/by-nc-nd/4.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2026-05-13T14:30:41Z",
    "title_canon_sha256": "965d6650c7305554093d64c14df732b393b9e6c0db1a64ced164742a9d2176c8"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2605.13596",
    "kind": "arxiv",
    "version": 1
  }
}