Pith Number

pith:EZH63GUI

pith:2020:EZH63GUIZJN25JRIYI2IJLLVPQ

not attested not anchored not stored refs resolved

CodeBLEU: a Method for Automatic Evaluation of Code Synthesis

Ambrosio Blanco, Daya Guo, Duyu Tang, Long Zhou, Ming Zhou, Neel Sundaresan, Shuai Lu, Shuai Ma, Shujie Liu, Shuo Ren

CodeBLEU evaluates generated code by adding syntax tree and data-flow matches to n-gram overlap so that scores align better with human programmer judgments than BLEU or exact accuracy.

arxiv:2009.10297 v2 · 2020-09-22 · cs.SE · cs.CL

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{EZH63GUIZJN25JRIYI2IJLLVPQ}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

Experimental results show that our proposed CodeBLEU can achieve a better correlation with programmer assigned scores compared with BLEU and accuracy.

C2weakest assumption

That the weighted combination of n-gram, AST, and data-flow matches will reliably reflect human judgment of code quality across tasks without the weights being overfitted to the specific evaluation sets.

C3one line summary

CodeBLEU improves correlation with human programmer scores on code synthesis tasks by adding syntactic AST matching and semantic data-flow matching to the standard BLEU n-gram approach.

References

93 extracted · 93 resolved · 16 Pith anchors

[1] Advances in Neural Information Processing Systems , pages=

[2] Achieving human parity on automatic chinese to english news translation · arXiv:1803.05567

[3] Unsupervised Neural Machine Translation · arXiv:1710.11041

[4] Unsupervised machine translation using monolingual corpora only · arXiv:1711.00043

[5] Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , volume=

Formal links

2 machine-checked theorem links

Cited by

48 papers in Pith

Guidelines for Empirical Studies in Software Engineering involving Large Language Models

TransAgent: Enhancing LLM-Based Code Translation via Fine-Grained Execution Alignment

CODEPROMPTZIP: Code-specific Prompt Compression for Retrieval-Augmented Generation in Coding Tasks with LMs

Automated Knowledge Component Generation for Interpretable Knowledge Tracing in Coding Problems

XOXO: Stealthy Cross-Origin Context Poisoning Attacks against AI Coding Assistants

Receipt and verification

First computed	2026-05-17T23:39:21.880910Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

264fed9a88ca5baea628c23484ad757c25db12adec91b448e72ca34c273020b2

Aliases

arxiv: 2009.10297 · arxiv_version: 2009.10297v2 · doi: 10.48550/arxiv.2009.10297 · pith_short_12: EZH63GUIZJN2 · pith_short_16: EZH63GUIZJN25JRI · pith_short_8: EZH63GUI

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/EZH63GUIZJN25JRIYI2IJLLVPQ \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 264fed9a88ca5baea628c23484ad757c25db12adec91b448e72ca34c273020b2

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "ffc5ae0ba7af0397ab2e62a8724894df8b10ba7e4d47095d5750799d25a2833a",
    "cross_cats_sorted": [
      "cs.CL"
    ],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.SE",
    "submitted_at": "2020-09-22T03:10:49Z",
    "title_canon_sha256": "034c637e9bb91e2a5a9b62931b450fe548591fef10be3c35891837159640f111"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2009.10297",
    "kind": "arxiv",
    "version": 2
  }
}