Pith Number

pith:4IPOKMJK

pith:2019:4IPOKMJKMNFBR6KT3HSUEQOANU

not attested not anchored not stored refs resolved

CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

Hamel Husain, Ho-Hsiang Wu, Marc Brockschmidt, Miltiadis Allamanis, Tiferet Gazit

Releasing the CodeSearchNet Corpus of 6 million functions and a challenge with 99 annotated queries enables evaluation of semantic code search across six languages.

arxiv:1909.09436 v3 · 2019-09-20 · cs.LG · cs.IR · cs.SE · stat.ML

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{4IPOKMJKMNFBR6KT3HSUEQOANU}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

To enable evaluation of progress on code search, we are releasing the CodeSearchNet Corpus and are presenting the CodeSearchNet Challenge, which consists of 99 natural language queries with about 4k expert relevance annotations of likely results from CodeSearchNet Corpus.

C2weakest assumption

The assumption that mechanically scraped and preprocessed function documentation yields sufficiently accurate and representative natural-language queries, and that the expert annotations are consistent and unbiased measures of relevance.

C3one line summary

Releases a large multi-language code corpus and expert-annotated challenge to benchmark semantic code search.

References

26 extracted · 26 resolved · 1 Pith anchors

[1] Miltiadis Allamanis. 2018. The Adverse Effects of Code Duplication in Machine Learning Models of Code. arXiv preprint arXiv:1812.06469 (2018) 2018

[2] Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. 2018. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR) 51, 4 (2018), 81 2018

[3] Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A Convolutional Attention Network for Extreme Summarization of Source Code. In Proceedings of the International Conference on Machine Learning 2016

[4] code2seq: Generating Sequences from Structured Representations of Code 2018 · arXiv:1808.01400

[5] A parallel corpus of Python functions and documentation strings for automated code documentation and code generation 2017 · arXiv:1707.02275

Formal links

2 machine-checked theorem links

Cited by

81 papers in Pith

From Execution to Education: A Bloom-Aligned Framework for Measuring Educational Control in LLMs

Bidirectional Small-Granularity Search between Code and Text

JupOtter: Cell-Level Bug Detection in Jupyter Notebooks

ConcernBERT: Learning Responsibilities Using Class Membership

Repository-Level Solidity Code Generation with Large Language Models: From Prompting to Fine-Tuning

Receipt and verification

First computed	2026-07-05T01:08:24.849922Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

e21ee5312a634a18f953d9e54241c06d2ac2a8a438558fe4bbe6d0847e2c3071

Aliases

arxiv: 1909.09436 · arxiv_version: 1909.09436v3 · doi: 10.48550/arxiv.1909.09436 · pith_short_12: 4IPOKMJKMNFB · pith_short_16: 4IPOKMJKMNFBR6KT · pith_short_8: 4IPOKMJK

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/4IPOKMJKMNFBR6KT3HSUEQOANU \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: e21ee5312a634a18f953d9e54241c06d2ac2a8a438558fe4bbe6d0847e2c3071

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "33e2d50bac7613c65aff732583ff944c4a2428b24e64c4b1b2072a205f86e5cc",
    "cross_cats_sorted": [
      "cs.IR",
      "cs.SE",
      "stat.ML"
    ],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.LG",
    "submitted_at": "2019-09-20T11:52:45Z",
    "title_canon_sha256": "30eb13f7e19b295d94df00528fcc5f3e2875ebea204bb9870398d335e448954e"
  },
  "schema_version": "1.0",
  "source": {
    "id": "1909.09436",
    "kind": "arxiv",
    "version": 3
  }
}