Pith Number

pith:Q7UB5XFR

pith:2025:Q7UB5XFR7FQCLRXIRFZJRIBGCW

not attested not anchored not stored refs pending

Training Language Models to Use Prolog as a Tool

Lukas Galke Poech, Niklas Mellgren, Peter Schneider-Kamp

Training language models to use Prolog as a tool uncovers a trade-off where reward focus on correctness yields higher accuracy but delegates reasoning to natural language, while symbolic rewards enforce auditable full programs at lower peak

arxiv:2512.07407 v3 · 2025-12-08 · cs.CL

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{Q7UB5XFR7FQCLRXIRFZJRIBGCW}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

We identify an accuracy--auditability trade-off: configurations tuned for correctness alone learn to delegate reasoning to natural language and use Prolog only for the final computation, while configurations rewarded for symbolic structure produce fully auditable programs at a cost in accuracy.

C2weakest assumption

That the observed behavioral difference between reward compositions is caused primarily by the reward signals rather than by model size limits, prompt engineering details, or the specific Prolog execution environment.

C3one line summary

Fine-tuning Qwen2.5-3B with GRPO on GSM8K to use Prolog yields competitive zero-shot MMLU performance but exposes an accuracy-auditability trade-off interpreted as reward hacking.

Formal links

1 machine-checked theorem link

Receipt and verification

First computed	2026-06-26T01:15:47.845834Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

87e81edcb1f96025c6e8897298a02615be91c17c1fec92947ff2bcd5f8514e68

Aliases

arxiv: 2512.07407 · arxiv_version: 2512.07407v3 · doi: 10.48550/arxiv.2512.07407 · pith_short_12: Q7UB5XFR7FQC · pith_short_16: Q7UB5XFR7FQCLRXI · pith_short_8: Q7UB5XFR

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/Q7UB5XFR7FQCLRXIRFZJRIBGCW \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 87e81edcb1f96025c6e8897298a02615be91c17c1fec92947ff2bcd5f8514e68

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "89a3e8783444e4545f3861edc16aa0a7516999393d1b0d9d54a7b29f9f9a1d73",
    "cross_cats_sorted": [],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2025-12-08T10:39:38Z",
    "title_canon_sha256": "8d5fb81c3dc72e44c8c988bdb8d7ede54bf789cea04382f0e3c10503335c8ca8"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2512.07407",
    "kind": "arxiv",
    "version": 3
  }
}