pith:UVBEQKV3
Guidelines for Empirical Studies in Software Engineering involving Large Language Models
Guidelines organize LLM use in software engineering studies into seven types and provide eight reporting rules to boost reproducibility.
arxiv:2508.15503 v6 · 2025-08-21 · cs.SE
Add to your LaTeX paper
\usepackage{pith}
\pithnumber{UVBEQKV3TMV4RHGB7ZAS4EUW2I}
Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge
Record completeness
Claims
We address this challenge through a collaborative effort of 22 researchers, presenting a taxonomy of seven study types that organizes how LLMs are used in SE research, together with eight guidelines for designing and reporting such studies.
The guidelines assume that declaring usage, reporting versions and prompts, human validation, and open baselines will sufficiently mitigate threats from non-determinism and opacity, without the paper providing new empirical evidence that these practices improve reproducibility outcomes.
A group of 22 researchers proposes seven study types and eight guidelines for empirical software engineering studies involving LLMs to enhance reproducibility and replicability.
Formal links
Cited by
Receipt and verification
| First computed | 2026-05-25T02:01:06.475903Z |
|---|---|
| Builder | pith-number-builder-2026-05-17-v1 |
| Signature | Pith Ed25519
(pith-v1-2026-05) · public key |
| Schema | pith-number/v1.0 |
Canonical hash
a542482abb9b2bc89cc1fe412e1296d2142692dcd147e588f2f07e6db6ae6680
Aliases
· · · · ·Agent API
Verify this Pith Number yourself
curl -sH 'Accept: application/ld+json' https://pith.science/pith/UVBEQKV3TMV4RHGB7ZAS4EUW2I \
| jq -c '.canonical_record' \
| python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: a542482abb9b2bc89cc1fe412e1296d2142692dcd147e588f2f07e6db6ae6680
Canonical record JSON
{
"metadata": {
"abstract_canon_sha256": "8dfd1ca1acf1d61e1b26adcb9bc4f84852bcfbebced2dc224ab91ec70d040fb3",
"cross_cats_sorted": [],
"license": "http://creativecommons.org/licenses/by/4.0/",
"primary_cat": "cs.SE",
"submitted_at": "2025-08-21T12:30:30Z",
"title_canon_sha256": "e0309463248c4b33dc5e30dc42b44758a0eef593f3d393d909aca9647c5e61ed"
},
"schema_version": "1.0",
"source": {
"id": "2508.15503",
"kind": "arxiv",
"version": 6
}
}