Pith Number

pith:ODXWRQYN

pith:2024:ODXWRQYNXKOR3ZX6FVYSFXZLUC

not attested not anchored not stored refs resolved

A Survey on Knowledge Distillation of Large Language Models

Can Xu, Chongyang Tao, Dacheng Tao, Jinyang Li, Ming Li, Reynold Cheng, Tao Shen, Tianyi Zhou, Xiaohan Xu

Knowledge distillation transfers advanced capabilities from proprietary LLMs like GPT-4 to open-source models such as LLaMA and Mistral.

arxiv:2402.13116 v4 · 2024-02-20 · cs.CL

Open paper page JSON Open Graph Bundle Merged state What is a Pith Number?

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

KD emerges as a pivotal methodology for transferring advanced capabilities from leading proprietary LLMs, such as GPT-4, to their open-source counterparts like LLaMA and Mistral, while also enabling model compression and self-improvement.

C2weakest assumption

That data augmentation within the KD framework can reliably enable open-source models to approximate the contextual adeptness, ethical alignment, and deep semantic insights of proprietary models.

C3one line summary

A comprehensive survey of knowledge distillation for LLMs structured around algorithms, skill enhancement, and vertical applications, highlighting data augmentation as a key enabler.

References

300 extracted · 300 resolved · 25 Pith anchors

[1] Advances in Neural Information Processing Systems , volume=

[2] arXiv preprint arXiv:2304.14233 , year=

[3] arXiv preprint arXiv:2305.07402 , year=

[4] arXiv preprint arXiv:2212.10192 , year=

[5] The Eleventh International Conference on Learning Representations , year=

Formal links

3 machine-checked theorem links

Cited by

22 papers in Pith

CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment

CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation

LLM-AutoDP: Automatic Data Processing via LLM Agents for Model Fine-tuning

DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection

Multi-Rollout On-Policy Distillation via Peer Successes and Failures

Receipt and verification

First computed	2026-05-17T23:37:42.480803Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

70ef68c30dba9d1de6fe2d7122df2ba0b9441751d771fab7e32eafc994127e60

Aliases

arxiv: 2402.13116 · arxiv_version: 2402.13116v4 · doi: 10.48550/arxiv.2402.13116 · pith_short_12: ODXWRQYNXKOR · pith_short_16: ODXWRQYNXKOR3ZX6 · pith_short_8: ODXWRQYN

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/ODXWRQYNXKOR3ZX6FVYSFXZLUC \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 70ef68c30dba9d1de6fe2d7122df2ba0b9441751d771fab7e32eafc994127e60

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "d3883b76b02e2cc11bb101e4b4b99096c4653e9dddd62c8cc77bd4da17007bbb",
    "cross_cats_sorted": [],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2024-02-20T16:17:37Z",
    "title_canon_sha256": "12f954505e87a6800491342b9b9738cfea9318a753e579e6b7d8f80c95b99e9c"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2402.13116",
    "kind": "arxiv",
    "version": 4
  }
}