Empirical Prompt Engineering for Construct Identification with Large Language Models

Brittney Hernandez; Claudia Ventura; Kylie L. Anglin; Stephanie Milan

REVIEW 4 major objections 5 minor 4 references

The paper claims that, for LLM classification of psychological constructs, empirically selecting prompt wording on labeled training data is the dominant driver of agreement with human expert codes, and that this can be formalized as a pract

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 18:41 UTC pith:BXFDN5DF

load-bearing objection The core finding—baseline wording matters more than personas, CoT, or explanations—is credible and practically useful, but the headline improvement magnitudes are inflated by selecting the best and worst of 50 random prompts on ~140 training texts. the 4 major comments →

arxiv 2512.03818 v3 pith:BXFDN5DF submitted 2025-12-03 cs.CL

Empirical Prompt Engineering for Construct Identification with Large Language Models

Kylie L. Anglin , Stephanie Milan , Brittney Hernandez , Claudia Ventura This is my paper

classification cs.CL

keywords prompt engineeringlarge language modelstext classificationpsychological constructshuman-machine alignmentfew-shot learningcodebook-guided promptsconstruct identification

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that, for LLM-based identification of psychological constructs, the wording of the baseline prompt is the main lever on agreement with human expert codes. Across three constructs and two language models, the authors find that empirically picking the best of many prompt variants in a labeled training set substantially improves alignment, with F1 gaps between top and bottom variants as large as 0.33 and relative gains up to 75 percent on high-inference constructs. Additive techniques such as personas, chain-of-thought, and explanations offer little or no consistent benefit, while well-chosen few-shot examples help mainly when the baseline prompt is poor. The paper recommends generate-and-test: create many prompt variants, score them on training data, select the best, and validate on hold-out data. If right, it gives applied researchers a concrete protocol for raising classification validity without waiting for thousands of hand-coded examples.

Core claim

The central claim is that prompt selection is better understood as an empirical search problem than as a matter of clever prompting tricks. The authors show that random combinations of construct definitions, task instructions, and inclusion/exclusion criteria, scored on a training set and validated on a held-out test set, produce the classifications most aligned with expert judgments, and that this holds across constructs and models. The best final prompts combined codebook-guided human-generated baselines, automatic prompt engineering, and few-shot examples. In contrast, persona, chain-of-thought, and explanation techniques produced small or inconsistent gains, mostly when the underlying pr

What carries the argument

The engine of the study is a combinatorial prompt space: five variants of a construct definition, five task instructions, and inclusion/exclusion criteria are randomly recombined into 50 baseline prompts, with iterative automatic prompt engineering generating further variants and random example sets providing few-shot candidates. F1 score on the training dataset is the selection criterion; the top and bottom prompts are then carried into development and hold-out testing. This machinery converts prompt wording from a craft into a parameterized space that can be brute-force searched, and the selection step is what the authors argue produces the large alignment gains.

Load-bearing premise

The load-bearing premise is that a training set of roughly 140 texts per construct is large enough that ranking dozens of prompt variants by F1 separates genuinely better prompts from random noise; if selection is mostly fitting noise, the reported gains will not reproduce on new data.

What would settle it

One concrete check: repeat the same 50-prompt generation and training-set selection, then score all 50 prompts on several fresh held-out samples; if the top training prompt no longer beats the median prompt on the hold-out sets, or if a randomly chosen prompt matches it, the selection effect is not real.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If the central claim is right, researchers can improve LLM-expert agreement by generating and testing many prompt variants rather than relying on a single intuitively written prompt.
Because additive techniques added little, practitioners can save computational cost and model run-time by skipping personas, chain-of-thought, or explanations and focusing on construct definitions, task instructions, and examples.
Few-shot examples should be seen as a repair tool for weak prompts: they can bring a poor prompt to near-parity with a good zero-shot prompt, but they add little to an already strong baseline.
The recommended protocol, train selection, development comparison, and hold-out evaluation, provides a template for high-stakes classification where LLM output needs validation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

A reasonable extension is that the empirically selected winning prompt may itself reveal which definition wording is clearest, and could feed back into human codebook development; the paper does not make this claim.
Because the protocol requires a labeled training set of roughly 50–100 texts, the approach is most feasible where some expert coding already exists; for entirely new constructs with no labeled data, the paper's recommendation cannot be applied directly.
The observed top-bottom gap likely overstates the average value of prompt variation, since selecting the most extreme of 50 random variants on a small training set is subject to overfitting; real-world gains on new texts may be smaller but still positive.
Combining this selection procedure with model-agnostic evaluation could make it a general quality-control step for LLM annotation in other expert domains, not just psychology.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The core finding—baseline wording matters more than personas, CoT, or explanations—is credible and practically useful, but the headline improvement magnitudes are inflated by selecting the best and worst of 50 random prompts on ~140 training texts.

read the letter

Worth a look, with a grain of salt. The paper's central claim is the right one: for psychological construct classification, the baseline prompt—construct definition, task framing, and examples—matters far more than the flashy additive techniques. That conclusion is consistent across three constructs and two models, and it's genuinely useful for applied researchers who are tired of generic prompt-engineering advice. The design is a real step up from most of this literature: participant-level train/dev/test splits, bootstrap standard errors, thousands of prompt evaluations, and released code. The authors also deserve credit for writing an honest limitations section—they explicitly acknowledge the null results could be Type II error and that generalizability is uncertain.

The soft spot is the magnitude of the claimed gains. The top and bottom baselines are the extremes of 50 randomly assembled human prompts, selected on training sets of only 135–143 texts. That makes the best-vs-worst gap—up to 0.33 F1, "as much as 75%"—an extreme-order statistic, inflated by selection on noise. The paper never reports the held-out test performance of a random or median prompt, so the reader can't tell what the expected gain of the recommended protocol actually is. The dev-set significance asterisks also lack a described hypothesis test and no multiple-comparison correction is applied, and the dev set is used repeatedly for few-shot/example selection. None of this kills the direction of the finding, but the quantitative claims should be softened or better supported. I'd also flag that the final test table reports only the selected prompt's absolute scores, not a comparison that isolates the benefit of selection.

Minor issues: two of three datasets are unavailable, though the gratitude data is public and code is there; and Table 6 shows a weird drop for few-shot chain-of-thought on the bottom baseline (0.29) that deserves an explanation.

Bottom line: this is a serious empirical paper that deserves peer review, but it needs a statistical revision—report the full distribution of prompt performance, add an out-of-sample comparison to an average or random prompt, and fix the significance testing. I'd bring it to a reading group, and I'd cite it with the caveat that the headline effect sizes are probably optimistic.

Referee Report

4 major / 5 minor

Summary. This manuscript reports three case studies of prompt engineering for LLM-based text classification of psychological constructs (gratitude, negative core beliefs, positive meaning making) using GPT-4 and Llama-3.3. The authors generate 50 codebook-guided human baseline prompts per construct, select the top and bottom prompts by F1 on a small training set, and then evaluate automatic prompt engineering, persona, chain-of-thought, explanation, and few-shot variants in a development set, with a final hold-out evaluation. The paper's central claim is that the baseline prompt—especially the construct definition and task framing—is the most determinative component of alignment with human codes, and that empirically selecting prompt variants substantially improves performance, whereas additive techniques provide smaller and less consistent benefits. It concludes with a recommended protocol of generating and evaluating many prompt variants on a labeled training set and validating on a hold-out set.

Significance. If supported, the paper would offer a practical, systematic protocol for a domain where LLM annotation is increasingly used and where prompt sensitivity is a real threat to validity. The design has notable strengths: participant-level train/dev/test splits, bootstrap standard errors, a final hold-out evaluation, public analysis code, and the use of both closed and open models across constructs of varying inference difficulty. These are real virtues and make the paper more credible than a purely anecdotal prompt-engineering study. However, the headline magnitude claims ('as much as 75% improvement,' 'up to 0.33 F1') rest on comparing the two extremes of 50 randomly generated prompts selected on training sets of roughly 135–143 texts. Under sampling noise, the best-of-50 estimate is inflated relative to the expected gain from the recommended protocol, and the paper does not provide the necessary random/median baseline or selection-adjusted analysis. The central practical conclusion is therefore plausible but not yet established at the reported strength.

major comments (4)

[Methods, 'Codebook-Guided Human-Generated Prompts'; Table 2] The central magnitude claim rests on the gap between the top and bottom of 50 prompts selected by F1 on the training set (135–143 texts; Table 2). Under the null that all prompts have equal true performance, the maximum of 50 noisy F1 estimates is expected to be well above the mean, so the reported 'up to 0.33 F1' and 'as much as 75%' (Introduction; Results) are inflated as estimates of the value of the recommended selection protocol. The paper should report the held-out distribution of F1 for all 50 baseline prompts, or compare the selected prompt against a randomly chosen or median prompt, and ideally provide a selection-adjusted estimate (e.g., repeated split-half selection or cross-validation) before claiming that empirical selection 'substantially improves' alignment.
[Methods, 'Evaluation'; Tables 3–8] The asterisks in Tables 3–8 are said to denote statistically significant differences, but no pairwise test is specified. Bootstrapped standard errors for single F1 estimates do not by themselves establish significance of a difference, and the development set is reused for selecting few-shot examples, personas, two chain-of-thought reasonings, two explanation wordings, and automatic-prompt-engineering candidates. The repeated comparisons are therefore not independent, and the reported significance levels do not reflect the full selection procedure. The paper should define the test (e.g., a bootstrap paired test or percentile interval), and either adjust for the number of comparisons or present these tables as descriptive with appropriate uncertainty intervals.
[Results, 'Final Performance in Testing Data'; Table 9] Table 9 gives a single point estimate of F1 for the final selected prompt, but no comparison with a random or average prompt from the same generation distribution. Since the protocol being recommended is 'generate as many variants as feasible and select,' the relevant quantity is the expected out-of-sample improvement of the selection rule, not the test performance of one realized optimum. Without the distribution of test F1 across the 50 baseline prompts, the reader cannot tell whether the selected prompt beats an arbitrary reasonable prompt. The column structure of Table 9 ('Bottom Baseline / Top Baseline / Zero-Shot / Few-Shot') is also under-specified and should be re-labeled.
[Discussion, protocol recommendation (steps 2–5); Limitations] The recommendation to generate as many prompt variants as feasible and select on a training set assumes a positive expected gain after selection. A single fixed train/dev/test split cannot validate this assumption, because the final prompt is chosen after repeated use of the development set. The authors should provide a repeated resampling analysis (e.g., multiple train/dev/test splits or nested cross-validation over the prompt-generation procedure) to estimate the expected gain and its variability. Without such an analysis, the reliability of the recommended protocol is unquantified, and the limitation discussion's acknowledgment of a 'moderately-sized labelled dataset' does not address the selection-bias issue directly.

minor comments (5)

[Methods, 'Evaluation'] The accuracy formula has a typo: the denominator 'TN+TP+FN+TP' should be 'TN+TP+FN+FP.' The F1 formula is also missing the factor 2 in the harmonic mean expression, although the final equality to 2TP/(2TP+FP+FN) is correct.
[Tables 1 and 2] Table 1 reports N = 600 for gratitude, but Table 2 sums to 575 (143 train + 287 dev + 145 test). Please clarify whether 25 cases were dropped or whether the table entries differ for another reason.
[Methods, 'Persona'] The third gratitude persona reads 'You an expert labeler'; this should presumably be 'You are an expert labeler.'
[Abstract and Introduction] Minor wording issues: 'theory-drive method' should be 'theory-driven method'; 'imminently scalable' should be 'eminently scalable.'
[Tables 3–8] The table notes state that asterisks indicate statistical significance, but the Methods section does not describe how p-values were computed. Please either add the test description to the Methods or change the table notes to describe the specific inference procedure used.

Circularity Check

0 steps flagged

No circularity: the paper is an empirical prompt-selection study whose headline results are evaluated on held-out development/test data, not on the data used to select prompts.

full rationale

This paper does not contain a derivation chain whose conclusions are equivalent to its inputs. The central claim is that empirically selecting prompt variants on a training dataset improves LLM-human alignment. The prompts are generated combinatorially, scored on a training set, and then re-evaluated on development and held-out testing sets (Methods: 'The training dataset was used to pilot prompts... We held our testing dataset until the final prompt and model for each concept was selected'). The top/bottom baseline distinction is defined by training-set F1, but the reported gaps and final performance are measured out-of-sample, so the result is not forced by construction. Automatic prompt engineering likewise selects candidates by measured F1 on training data, with final evaluation on the hold-out test set (Table 9), not by the LLM's self-assessment. The few-shot example selection is also validated against the remaining training texts and ultimately against dev/test sets. The only self-citation (Anglin 2024) supports the standard practice of keeping training, development, and testing sets independent; it is not load-bearing for any of the empirical conclusions. Concerns about selecting extreme order statistics from ~140 training texts and about repeated dev-set comparisons are legitimate threats to the magnitude or generalizability of the reported gains, but they are statistical overfitting/validity concerns, not circularity. Accordingly, no circular step is identified and the score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The paper introduces no mathematical axioms or invented entities. The listed free parameters are hand-chosen search-breadth hyperparameters rather than fitted model parameters; they shape the magnitude of the reported gains and the null results for additive techniques. The axioms are domain-level assumptions about ground truth, leakage, reliability thresholds, and generalizability.

free parameters (4)

Prompt search breadth (50 random variants per construct) = 50
The magnitude of the top-vs-bottom baseline gap and the claimed benefit of empirical selection are functions of how many random variants are generated and screened; more variants inflate the extremal gap. Chosen by hand, not fitted.
Automatic prompt engineering search depth (5 iterations × 5 candidates) = 5 iterations, 5 candidates/iteration
Determines how good the best automatically generated prompt can become; affects the conclusion that APE improves low-performing baselines.
Few-shot example search budget (50 combinations, 50-example pool, 1–10 examples) = 50 combos; pool 50; 1–10 examples
The observed few-shot benefit depends on searching 50 random example combinations and picking the best; a larger budget would likely improve few-shot results.
Additive technique search budgets (3 personas; 2 CoT reasonings; 2 explanation wordings) = 3; 2; 2
The null results for personas/CoT/explanations are conditional on small hand-chosen variant sets; wider searches might find effective versions of these techniques.

axioms (5)

domain assumption Human labels are the gold standard; F1 against human labels measures the validity of LLM construct identification.
Used as the evaluation criterion throughout; the authors note human labels carry their own error (inter-rater reliability paragraph) but treat them as ground truth.
domain assumption Cohen's Kappa of 0.70–0.82 is sufficient to treat human codes as reliable benchmarks.
Invoked when designating human codes as 'true' labels; no formal threshold justification is provided.
domain assumption The expressive-writing samples were not part of LLM pretraining, so no label leakage for negative core beliefs and meaning making.
Stated in Strengths and Limitations: the clinical texts are not public, so leakage is 'certainly not' present.
domain assumption Findings from three constructs and two models generalize to other psychological constructs and LLMs.
The paper's recommendations assume broader applicability; authors acknowledge this as a limitation.
domain assumption The random combinatorial procedure generates a representative sample of plausible prompt wordings.
Underlies the comparison of top vs bottom of 50 prompts; with small training sets, extremal selections are affected by noise.

pith-pipeline@v1.3.0-alltime-deepseek · 17135 in / 17873 out tokens · 146047 ms · 2026-08-03T18:41:44.708544+00:00 · methodology

0 comments

read the original abstract

Due to their architecture and vast pre-training data, large language models (LLMs) demonstrate strong performance on text classification tasks. However, LLM classifications are highly responsive to prompt wording, particularly, as we show, in domains like psychology, where constructs are often latent, complex, and theory driven. Here, we present and evaluate a systematic framework for improving psychological construct identification through prompt engineering. We combinatorially generate prompts by appending random selections of multiple variants of construct definitions, task instructions, coding guidance, and examples. Empirically selecting the highest performing of these combinations in a training dataset substantially improves alignment between LLM and human classifications. In contrast, prompting techniques such as personas, chain-of-thought reasoning, and explanations provide smaller and less consistent improvements. This finding holds across multiple models and constructs. Overall, the approach we describe offers a practical, systematic, and theory-aware method for increasing the alignment between human and LLM classifications in settings where validity is critical.

Figures

Figures reproduced from arXiv: 2512.03818 by Brittney Hernandez, Claudia Ventura, Kylie L. Anglin, Stephanie Milan.

**Figure 2.** Figure 2: Performance Evolution with Automatic Prompt Engineering, as Assessed in the [PITH_FULL_IMAGE:figures/full_fig_p022_2.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

4 extracted references · 3 linked inside Pith

[1]

Addressing threats to validity in supervised machine learning: A frame- work and best practices for education researchers

Anglin, K. (2024). “Addressing threats to validity in supervised machine learning: A frame- work and best practices for education researchers”. In:AERA Open10, pp. 1–21.doi: 10.1177/23328584241303495. Anil, Rohan et al. (2023). “Palm 2 technical report”. In:arXiv preprint arXiv:2305.10403. doi:https://doi.org/10.48550/arXiv.2305.10403. Battle, R. and T. G...

Pith/arXiv arXiv doi:10.48550/arxiv.2305.10403 2024
[9]

GPT is an effective tool for multilingual psychological text analysis

Radford, Alec et al. (2018).Improving language understanding by generative pre-training. Publisher: OpenAI. Rathje, Steve et al. (2024). “GPT is an effective tool for multilingual psychological text analysis”. In:Proceedings of the National Academy of Sciences121.34, e2308950121. Reinhold, M., P.-C. B¨ urkner, and H. Holling (2018). “Effects of expressive...

2018
[17]

A prompt pattern catalog to enhance prompt engineering with chatgpt

White, Jules et al. (2023). “A prompt pattern catalog to enhance prompt engineering with chatgpt”. In:arXiv preprint arXiv:2302.11382. Zhao, Z. et al. (2021). “Calibrate before use: Improving few-shot performance of language models”. In:International Conference on Machine Learning, pp. 12697–12706. Zhou, Y. et al. (2022). “Large language models are human-...

Pith/arXiv arXiv 2023
[80]

Prompt programming for large language mod- els: Beyond the few-shot paradigm

Reynolds, Laria and Kyle McDonell (2021). “Prompt programming for large language mod- els: Beyond the few-shot paradigm”. In:Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–7.doi:https://doi.org/10.48550/ arXiv.2102.07350. Sahoo, P. et al. (2024). “A systematic survey of prompt engineering in large language model...

Pith/arXiv arXiv doi:10.48550/arxiv.2102.07350 2021

[1] [1]

Addressing threats to validity in supervised machine learning: A frame- work and best practices for education researchers

Anglin, K. (2024). “Addressing threats to validity in supervised machine learning: A frame- work and best practices for education researchers”. In:AERA Open10, pp. 1–21.doi: 10.1177/23328584241303495. Anil, Rohan et al. (2023). “Palm 2 technical report”. In:arXiv preprint arXiv:2305.10403. doi:https://doi.org/10.48550/arXiv.2305.10403. Battle, R. and T. G...

Pith/arXiv arXiv doi:10.48550/arxiv.2305.10403 2024

[2] [9]

GPT is an effective tool for multilingual psychological text analysis

Radford, Alec et al. (2018).Improving language understanding by generative pre-training. Publisher: OpenAI. Rathje, Steve et al. (2024). “GPT is an effective tool for multilingual psychological text analysis”. In:Proceedings of the National Academy of Sciences121.34, e2308950121. Reinhold, M., P.-C. B¨ urkner, and H. Holling (2018). “Effects of expressive...

2018

[3] [17]

A prompt pattern catalog to enhance prompt engineering with chatgpt

White, Jules et al. (2023). “A prompt pattern catalog to enhance prompt engineering with chatgpt”. In:arXiv preprint arXiv:2302.11382. Zhao, Z. et al. (2021). “Calibrate before use: Improving few-shot performance of language models”. In:International Conference on Machine Learning, pp. 12697–12706. Zhou, Y. et al. (2022). “Large language models are human-...

Pith/arXiv arXiv 2023

[4] [80]

Prompt programming for large language mod- els: Beyond the few-shot paradigm

Reynolds, Laria and Kyle McDonell (2021). “Prompt programming for large language mod- els: Beyond the few-shot paradigm”. In:Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–7.doi:https://doi.org/10.48550/ arXiv.2102.07350. Sahoo, P. et al. (2024). “A systematic survey of prompt engineering in large language model...

Pith/arXiv arXiv doi:10.48550/arxiv.2102.07350 2021