Pith Number

pith:JIDEKWRO

pith:2022:JIDEKWROCPKMO3IDKLDMVYQ5HZ

not attested not anchored not stored refs resolved

PAL: Program-aided Language Models

Aman Madaan, Graham Neubig, Jamie Callan, Luyu Gao, Pengfei Liu, Shuyan Zhou, Uri Alon, Yiming Yang

LLMs generate programs as reasoning steps and let a Python interpreter execute them to solve math and symbolic problems more accurately than much larger models using chain-of-thought.

arxiv:2211.10435 v2 · 2022-11-18 · cs.CL · cs.AI

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{JIDEKWROCPKMO3IDKLDMVYQ5HZ}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

PAL using Codex achieves state-of-the-art few-shot accuracy on the GSM8K benchmark of math word problems, surpassing PaLM-540B which uses chain-of-thought by absolute 15% top-1.

C2weakest assumption

That the LLM will reliably generate correct, executable programs whose logic matches the intended reasoning without introducing its own coding or planning errors.

C3one line summary

PAL improves few-shot reasoning accuracy by having LLMs generate executable programs rather than text-based chains of thought, outperforming much larger models on math and logic benchmarks.

References

44 extracted · 44 resolved · 18 Pith anchors

[1] Do As I Can, Not As I Say: Grounding Language in Robotic Affordances 2022 · arXiv:2204.01691

[2] https://aclanthology.org/N19-1245 M ath QA : Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms 2019

[3] Giving bert a calculator: Finding operations and arguments with reading comprehension 1909

[4] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert - Voss, A., Krueger, G., Henighan, T., Child, R., Ram 2020

[6] Evaluating Large Language Models Trained on Code 2021 · arXiv:2107.03374

Formal links

2 machine-checked theorem links

Cited by

35 papers in Pith

AdaSwitch: Adaptive Switching between Small and Large Agents for Effective Cloud-Local Collaborative Learning

Enhancing the Code Reasoning Capabilities of LLMs via Consistency-based Reinforcement Learning

RepairAgent: An Autonomous, LLM-Based Agent for Program Repair

ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving

Llemma: An Open Language Model For Mathematics

Receipt and verification

First computed	2026-05-17T23:38:53.444657Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

4a06455a2e13d4c76d0352c6cae21d3e4ca1b18a921a1e557ad74cef254acadb

Aliases

arxiv: 2211.10435 · arxiv_version: 2211.10435v2 · doi: 10.48550/arxiv.2211.10435 · pith_short_12: JIDEKWROCPKM · pith_short_16: JIDEKWROCPKMO3ID · pith_short_8: JIDEKWRO

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/JIDEKWROCPKMO3IDKLDMVYQ5HZ \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 4a06455a2e13d4c76d0352c6cae21d3e4ca1b18a921a1e557ad74cef254acadb

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "abc4d6b463b55fea9aaebb576a3312e727cdfd12b19ac275fdcb5e59210a2932",
    "cross_cats_sorted": [
      "cs.AI"
    ],
    "license": "http://creativecommons.org/publicdomain/zero/1.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2022-11-18T18:56:13Z",
    "title_canon_sha256": "e6a3e4ab69f3f371b2c5bde4a1bf6463ca13fe3fbea0168f2641acc77ba1c924"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2211.10435",
    "kind": "arxiv",
    "version": 2
  }
}