Pith. sign in

REVIEW 4 major objections 5 minor 10 references

Prompt Engineering: How Prompt Vocabulary affects Domain Knowledge

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that LLM performance peaks inside a mid-range of prompt vocabulary specificity, not at maximum specificity.

desk verdict The paper's real finding is a null/negative controlled result about prompt specificity; its headline 'optimal specificity range' is a post-hoc descriptive artifact, not a tested design rule. read the letter →

arxiv 2505.17037 v1 pith:Y3DBRITT submitted 2025-05-10 cs.CL

Dimitri Schreiter This is my paper · ORCID
classification cs.CL
keywords promptspecificityengineeringsynonymizationWordNetlargelanguagemodelsdomainknowledgequestionansweringreasoningtasks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The thesis asks whether making the vocabulary of prompts more specific improves how large language models answer expert questions in STEM, law, and medicine. It builds a synonymization framework that swaps nouns, verbs, and adjectives for synonyms of measured low, intermediate, or high specificity, applied at three replacement rates to samples from MMLU, GPQA, and GSM8K. The central finding, consistent across all four tested models, is that performance is best inside a middle band of prompt specificity, roughly 17.7 to 19.7 for nouns and 8.1 to 10.6 for verbs, and that pushing beyond the band does not help nouns and significantly hurts verbs in reasoning tasks. If correct, the practical rule for prompt design is to tune specificity into the band rather than maximize it.

What carries the argument

The load-bearing mechanism is a five-step synonymization pipeline: part-of-speech tagging, word sense disambiguation (performed by Llama-3.1-70B-Instruct, measured at 0.79 accuracy), crawling of synonyms through WordNet hypernym and hyponym relations, assignment of continuous specificity scores, and replacement of words at 33, 67, and 100 percent rates. For nouns and verbs, specificity is computed as distance from the taxonomy root plus a normalized hyponym count minus a sense-count penalty; for adjectives, a new fractional measure is introduced based on counts of similar words, synonyms, antonyms, and senses. These scores convert coarse low/intermediate/high categories into a continuous prompt-level specificity value, which is what lets the paper locate the performance peaks as specific numerical bands.

What would settle it

Take the same prompts, correct the word senses by hand before synonymizing, and re-measure the accuracy-versus-specificity curves: if the reported optimal bands for nouns and verbs shift or disappear, the ranges were artifacts of sense errors rather than effects of specificity. A quicker check is to tabulate the disambiguation error rate per specificity category (low, intermediate, high) across the three datasets.

Watch

Extended reading notes

Core claim

Prompt specificity has a non-monotonic effect on LLM performance: for nouns and verbs there is an optimal specificity interval, shared across four differently sized and differently trained models, within which question-answering and reasoning accuracy peaks, with performance degrading or plateauing outside it. The claim is supported by a stepwise synonymization of sampled instructions, where prompts are assigned continuous specificity scores and correlated with accuracy. Beyond the optimal range, raising noun specificity leaves performance largely unchanged, while raising verb specificity in the reasoning benchmark (GSM8K) yields strong negative correlations, for example -0.89 for Llama-3.1-70B-Instruct and -0.87 for Mistral-Large 2. Only 5 of 24 model-dataset evaluations reach statistical significance, so the paper's headline negative result is concentrated in verbs on reasoning tasks; the paper presents the optimal band as the actionable discovery, correcting the intuition that maximally specific expert wording is always best.

Load-bearing premise

The pipeline assumes that the 21 percent of word senses the disambiguation step gets wrong only add random noise and do not systematically push wrong-sense synonyms into particular specificity categories.

Editorial extensions

If this is right

  • Prompt designers in specialized domains should target the mid-specificity band instead of maximizing term precision, since over-specific nouns do not help and over-specific verbs measurably degrade reasoning.
  • The optimal band's consistency across four models of different families and sizes suggests a shared property of how LLMs process prompt wording rather than a quirk of any single model.
  • Verb specificity is the more consequential lever: the significant negative correlations concentrate in verbs on the chain-of-thought math task, implying that rewording actions and relations disrupts stepwise reasoning more than rewording entities.
  • The WordNet-based specificity score offers a cheap, parameter-free diagnostic that could complement existing prompt-sensitivity methods in deciding when a prompt rewrite is worth testing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is variance: within the optimal band, model outputs may be more stable across seeds and paraphrases; if the band is where performance peaks and jitter drops, it becomes a genuine operating point rather than a statistical artifact.
  • The adjective-specificity equation, validated only in passing (median Spearman 0.50 against an LLM judge), may be the most portable piece: it could be tested on sentiment or concreteness benchmarks where adjectives carry the semantics, and if it survives there it becomes a general lexical tool.
  • Because the noun optimum (about 17.7 to 19.7) sits below the corpus mean specificity (21.37) while verbs cluster near their own mean, the band may track the lexical density of pretraining text; that predicts the band shifts for other languages or heavily technical corpora.
  • Read together with the mostly flat noun results, the findings suggest specificity is a cost curve with a flat middle: a prompt-selection heuristic that holds specificity inside the band while minimizing verb perturbation could be evaluated on held-out benchmarks as a practical optimization rule.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper develops a synonymization framework that replaces nouns, verbs, and adjectives in domain-specific prompts with synonyms of varying WordNet-based specificity, at replacement ratios of 33%, 67%, and 100%. It evaluates four LLMs (Llama-3.1-70B-Instruct, Granite-13B-Instruct-V2, Flan-T5-XL, and Mistral-Large 2) on MMLU, GPQA, and GSM8K tasks. The main reported findings are that increasing prompt specificity generally has little or negative impact on accuracy, and that there exists an 'optimal specificity range' (noun specificity roughly 17.7–19.7, verb specificity roughly 8.1–10.6) within which LLMs perform best. The paper also introduces an adjective-specificity measure and presents a GPT-4o-based validation of that measure.

Significance. If the optimal-range claim were supported, it would provide a practical, actionable rule for prompt design. The controlled manipulation experiments are a useful negative result: they suggest that blanket increases in specificity do not help and can hurt, especially for verbs in reasoning tasks. The framework is transparent, spans four models and three task formats, and includes a WSD validation step; this is a solid basis for follow-up work. However, the central design rule is not supported by the controlled experiments as reported, and the current evidence supports only a descriptive, exploratory statement about where correct answers concentrate on original prompts.

major comments (4)
  1. [Section 4, Q3; Fig. 10; Table 8] The claimed optimal specificity range is derived entirely from original, unmodified prompts, not from the specificity-manipulation experiments. The histograms in Fig. 10 are marginal associations: original questions with higher specificity may differ in domain, topic, or difficulty, so the concentration of correct answers near a given specificity value is confounded. The abstract's design rule ('manipulating prompts within this range could maximize LLM performance') would require a prospective test in which prompts are independently varied to fall inside versus outside the range; no such test is presented. At minimum, the claim should be reframed as an exploratory hypothesis, not a design rule.
  2. [Tables 8 and 9] The numeric 'optimal ranges' in Table 8 appear to be the minimum-to-maximum span of the nine prompt-specificity values produced by the synonymization grid in Table 9, not performance peaks estimated from accuracy. Table 9 reports specificity values per replacement ratio and specificity level but does not report any accuracy associated with those values, and its caption calls them 'Optimal Specificities' without a performance criterion. The abstract's 17.7–19.7 noun range mixes the Q1 KDE boundary (17.74) with one model's median (19.70), and Fig. 10's histogram modes are yet another set of numbers; the paper should specify exactly which statistic defines the optimal range and justify why that statistic identifies an optimum.
  3. [Section 4, Q2; Table 7; Figs. 8–9] The controlled manipulation experiments directly undercut the optimal-range claim. Table 7 shows that only 5 of 24 correlations are significant, and the significant correlations are negative, e.g., verb specificity in GSM8K for Llama-3.1-70B-Instruct (rho = -0.89, p = 5.4e-4) and Mistral-Large 2 (rho = -0.87, p = 0.001). No condition in the controlled experiments shows an interior performance peak inside the claimed optimal ranges. If the optimal-range claim is to be retained, the authors must explain why the controlled manipulation does not produce the predicted non-monotonic pattern, or they must soften the conclusion to a descriptive observation about original prompts.
  4. [Section 6 and Section 3.3] The WSD accuracy of 0.79 means that roughly one in five synonym choices may use the wrong sense, and the paper assumes this adds only noise. That assumption is load-bearing for Q2: if misidentified senses concentrate in particular specificity categories, the observed accuracy differences could reflect semantic corruption rather than specificity per se. The limitation is acknowledged in Section 6, but the manuscript should provide a sensitivity analysis, for instance by re-running the key correlations on a human-verified subset of synonym substitutions, before the null effect for nouns and the negative effect for verbs can be attributed to specificity.
minor comments (5)
  1. [Section 4, Q1] The text says 'Since all p-values are smaller than 0.05, we cannot reject the null hypothesis and therefore the distributions cannot be considered as normal distributions.' This is backwards: a p-value below 0.05 leads to rejection of the null hypothesis of normality, which is the intended conclusion. The wording should be corrected.
  2. [Section 2, Q2 (categorical approach)] The phrase 'precentral changes in accuracy' appears to be a typo; it should read 'percentage changes in accuracy'.
  3. [Section 3.1; Table 1] The dataset name 'MMMUL' appears once in Table 1; this should be 'MMLU'.
  4. [Figure 10 and Table 8] Figure 10's dashed line is described as marking 'the prompt specificity associated with the highest number of correct answers,' but Table 8 reports medians. The paper should clarify whether the optimal specificity is defined by the histogram mode, the median, or some other statistic, and use that definition consistently.
  5. [Section 3.3, adjective specificity] The adjective specificity equation is based on four equal-weight additive assumptions, and the GPT-4o validation reports a median Spearman correlation of 0.50 with 32 of 91 negative correlations. The paper should state more explicitly that this is a preliminary measure, and that the small adjective sample sizes (e.g., 4 GSM8K samples) preclude any conclusions about adjective effects on performance.

Circularity Check

2 steps flagged · score 6.0 of 10

The claimed 'optimal specificity range' is constructed from the experimental grid: Table 8 is the min/median/max of the nine prompt-specificity inputs in Table 9, renamed as a performance finding.

  1. fitted input called prediction [Section 4, Q3 Answer (Tables 8 and 9)]
    "We apply this approach to the modified instructions, examining each replacement level, specificity level, model, and dataset combination. The resulting prompt specificities for each parameter combination are displayed in Appendix B, Tab. 9. ... Based on this assessment, we derive the ranges of optimal prompt specificity for each model and part of speech, displayed in Tab. 8, where the LLM achieves the highest performance across all datasets."

    Table 8's Lower/Upper/Median values are exactly the minimum, median, and maximum of the nine prompt-specificity values in Table 9 ('All' rows), which are the independent-variable settings produced by the synonymization grid. For example, Granite-13b nouns: Table 8 gives 17.24 / 22.00 / 18.95, matching Table 9's All rows Low-33% = 17.23, High-100% = 22.00, and High-33% = 18.95 (the median of the nine values). The grid was deliberately built to span low-to-high specificity (Section 3.1, steps IV-V), so the 'optimal range' is just the range of the manipulated input values. Calling it 'where the LLM achieves the highest performance' renames the experimental design as an empirical discovery; no performance metric enters Table 9 or the construction of Table 8.

  2. self definitional [Section 4, Q3 (Figure 10 analysis)]
    "In each histogram, the x-axis represents the prompt specificity for the respective part of speech, while the y-axis indicates the count of correct answers. A dashed line marks the prompt specificity associated with the highest number of correct answers for each model. ... From these observations, we can infer that for the original, unmodified samples, a moderate level of specificity improves model performance."

    The 'optimal' specificity is defined as the bin with the highest count of correct answers, i.e., the mode of the correct-answer distribution. Inferring that 'a moderate level of specificity improves model performance' from the location of that mode restates the definition: every histogram has a mode, and labeling the mode as 'optimal' adds no independent performance comparison. The controlled manipulation (Q2, Table 7) mostly showed null or negative effects, so the Q3 marginal histogram mode cannot support the causal design rule; it is a descriptive summary of the original-prompt data by construction.

full rationale

The paper's negative verb-specificity correlations (Table 7) are genuine empirical findings from controlled synonym manipulation, and the adjective-specificity measure is benchmarked against GPT-4o, so those parts are not circular. The WSD pipeline cites prior work from the same research group (Wahle et al., 2021), but it is benchmarked in Table 3 and is not the load-bearing step for the central claim. The central design rule, however, is circular in its construction: the 'optimal specificity ranges' in Table 8 are the min/median/max of the prompt-specificity grid in Table 9 — i.e., the range of the experimental inputs — rather than any performance-derived optimum. The Q3 histogram analysis used only original, unmodified prompts, so the 'optimal' range is the mode of the correct-answer distribution, a tautological label rather than a prospective test. The abstract's 17.7–19.7 range mixes the Q1 KDE boundary (17.74) with one model's median (19.70), further indicating that the claimed range was assembled from the framework's own inputs rather than from a performance peak. Overall, the central 'optimal specificity range' reduces by construction to the experimental grid and the histogram mode, warranting a partial circularity score of 6.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The results depend on WordNet's structure as a model of specificity, on WSD accuracy (79%), on the assumption that synonym substitution preserves meaning, and on the causal interpretation of a descriptive mode. The adjective specificity measure rests on four untested equal-weight assumptions. No new physical or conceptual entities are postulated.

free parameters (3)
  • Noun specificity category boundaries = low-intermediate: 17.74; intermediate-high: 20.90
    KDE intersection points computed on the same noun specificity distribution used to label synonyms low/intermediate/high (Section 4, Q1). These labels feed all categorical accuracy comparisons.
  • Verb specificity category boundaries = low-intermediate: 11.17; intermediate-high: 13.74
    Same KDE procedure for verbs; the resulting categories define the replacement specificity levels in the experiments.
  • Adjective specificity weights (ssw, s, a, l) = 1, 1, 1, 1
    Equation 2 assigns equal unit weight to the four inverse-proportionality terms; no fitting or sensitivity analysis is reported, and validation against GPT-4o gives median Spearman 0.50.
assumptions (5)
  • domain assumption WordNet hypernym/hyponym depth and hyponym counts are valid proxies for semantic specificity.
    The noun/verb specificity score (Eq. 1) and all category definitions inherit this from Bolognesi et al. (2020); the paper does not independently validate it.
  • domain assumption The 21% WSD misclassification rate (accuracy 0.79) adds only noise and does not systematically bias specificity comparisons.
    Section 3.3 and 6; if wrong-sense synonyms cluster in certain specificity levels, results would be confounded.
  • ad hoc to paper Adjective specificity is inversely proportional to ssw, s, a, and l, with equal weights and additive combination.
    Section 3.3 states these four assumptions and the additive model; they are not derived or sensitivity-tested.
  • ad hoc to paper The mode of the correct-answer distribution on original prompts identifies a causal optimal specificity range.
    Q3 and Fig. 10 interpret the histogram of correct answers as an optimal range for prompt design, assuming the correlation is causal.
  • domain assumption Substituting a same-sense synonym preserves the task meaning and difficulty of the prompt.
    The framework's design (Section 3.1) assumes meaning is preserved so that accuracy changes can be attributed to specificity; WSD errors and connotation shifts weaken this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Prompt Engineering: How Prompt Vocabulary affects Domain Knowledge." pith.science (2026). https://pith.science/paper/Y3DBRITT

@misc{pith2026250517037,
  author       = {Pith},
  title        = {Pith review of: Prompt Engineering: How Prompt Vocabulary affects Domain Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y3DBRITT}},
  note         = {Machine review of arXiv:2505.17037}
}
read the original abstract

Prompt engineering has emerged as a critical component in optimizing large language models (LLMs) for domain-specific tasks. However, the role of prompt specificity, especially in domains like STEM (physics, chemistry, biology, computer science and mathematics), medicine, and law, remains underexplored. This thesis addresses the problem of whether increasing the specificity of vocabulary in prompts improves LLM performance in domain-specific question-answering and reasoning tasks. We developed a synonymization framework to systematically substitute nouns, verbs, and adjectives with varying specificity levels, measuring the impact on four LLMs: Llama-3.1-70B-Instruct, Granite-13B-Instruct-V2, Flan-T5-XL, and Mistral-Large 2, across datasets in STEM, law, and medicine. Our results reveal that while generally increasing the specificity of prompts does not have a significant impact, there appears to be a specificity range, across all considered models, where the LLM performs the best. Identifying this optimal specificity range offers a key insight for prompt design, suggesting that manipulating prompts within this range could maximize LLM performance and lead to more efficient applications in specialized domains.

Figures

Figures reproduced from arXiv: 2505.17037 by the authors.

Figure 1
Figure 1. Specificity-based Synonymization Framework. Representation of the specificity-based syn￾onymization framework used to synonymize the prompt instructions with varying specificities of all datasets. The preprocessing includes five key steps starting with the retrieval of parts (I) of speech from the original instruction, crawling synonyms (II) and calculate the specificity scores (III) for all parts of speech (the gre… view at source ↗
Figure 2
Figure 2. Confusion Matrix of WSD Model Eval￾uation. The performance agreement of Llama-3.1- 70B-Instruct and finetuned T5 for WSD when pre￾dicting against the human evaluated ground truth. Model Accuracy Fine-Tuned T5 0.63 ± 0.05 Llama-3.1-70B-Instruct 0.79 ± 0.02 [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Spearman-Correlations for Adjective Ranking. The histogram displays the distribution of Spearman correlations from the LLM-as-a-judge experiment, which compares the model’s qualita￾tive ranking of adjective specificity with the calcu￾lated specificity score measure for 91 samples. The median Spearman correlation in this distribution is 0.50. The model generated an ordered list where the first element represents the … view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: Example for Prompt Specificity Calcu￾lation. This example schematically illustrates the calculation of the prompt specificity, by aggregat￾ing the specificities of one part of speech (nouns in this case) and calculating the average that we call prompt specificity. Addi…
Figure 5
Figure 5. Figure 5: Specificity Score Distribution for nouns, verbs and adjectives. The histograms show the distribution of specificity scores for the respective part of speech. The mean specificity score for nouns is µnouns = 21.37, for verbs µverbs = 12.09 and for adjectives µadjectives…
Figure 6
Figure 6. Figure 6: Specificity Score Distributions for nouns and verbs according to specificity. The histogram represents the distribution, including Kernel-Density-Estimations, of the specificity score for nouns and verbs according to their specificity. Based on the intersection points …
Figure 7
Figure 7. Figure 7: Average Accuracy Comparison across multiple LLMs for Nouns. Average accuracy comparison across all models (Granite-13B-Instruct-v2, Flan-T5-XL, LLaMA-3.1-70B-Instruct, and Mistral-Large 2) for the datasets (MMLU, GSM8K, GPQA) for varying specificity levels (low, interm…
Figure 8
Figure 8. Figure 8: Average Accuracy across all Datasets and Models for Nouns. These heatmaps represent the performance differences in percentages for each model across all data for nouns. Each x-axis represents the replacement ratio (fraction of how many synonyms were used), the y-axis e…
Figure 9
Figure 9. Figure 9: Average Accuracy across all Datasets and Models for Verbs. These heatmaps represent the performance differences in percentages for each model across all data for verbs. Each x-axis represents the replacement ratio (fraction of how many synonyms were used), the y-axis e…
Figure 10
Figure 10. Figure 10: Distribution of correct Answers for each Model. These histograms represent the distributions of correct answers for the original samples for each model and part of speech. The x-axis depicts the re￾spective prompt specificity of the unmodified instructions, while the …
Figure 11
Figure 11. Figure 11: Prompt Templates. Overview of all the prompt templates used for performing the question￾answering and reasoning tasks (MMLU, GPQA and GSM8K), and the LLM-as-a-Judge experiment for the adjective specificity measure evaluation [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]
Figure 12
Figure 12. Figure 12: Average Accuracy Comparison across multiple LLMs for Adjectives. Average accuracy comparison across all models (Granite-13B-Instruct-v2, Flan-T5-XL, LLaMA-3.1-70B-Instruct, and Mistral￾Large 2) for the datasets (MMLU, GSM8K, GPQA) for original and 100% high specificit…
Figure 13
Figure 13. Figure 13: Average Accuracy Comparison across multiple LLMs for Verbs. Average accuracy comparison across all models (Granite-13B-Instruct-v2, Flan-T5-XL, LLaMA-3.1-70B-Instruct, and Mistral-Large 2) for the datasets (MMLU, GSM8K, GPQA) for varying specificity levels (low, inter…
Figure 14
Figure 14. Figure 14: Example for Task686 (MMLU). An example from the MMLU dataset: ”Task686”. This question is from the biology domain, which requires domain knowledge at a college level. 27 [PITH_FULL_IMAGE:figures/full_fig_p029_14.png]
Figure 15
Figure 15. Figure 15: Example for Task687 (MMLU). An example from the MMLU dataset: ”Task687”. This question is from the chemistry domain, which requires domain knowledge at a college level. MMLU: Task688 Which of the following comes closest to being a perfectly secure encryption scheme? (…
Figure 16
Figure 16. Figure 16: Example for Task688 (MMLU). An example from the MMLU dataset: ”Task688”. This question is from the computer science domain, which requires domain knowledge at a college level. 28 [PITH_FULL_IMAGE:figures/full_fig_p030_16.png]
Figure 17
Figure 17. Figure 17: Example for Task689 (MMLU). An example from the MMLU dataset: ”Task689”. This question is from the mathematics domain, which requires domain knowledge at a college level. MMLU: Task691 A resistor in a circuit dissipates energy at a rate of 1 W. If the voltage across t…
Figure 18
Figure 18. Figure 18: Example for Task691 (MMLU). An example from the MMLU dataset: ”Task691”. This question is from the physics domain, which requires domain knowledge at a college level. 29 [PITH_FULL_IMAGE:figures/full_fig_p031_18.png]
Figure 19
Figure 19. Figure 19: Example for Task699 (MMLU). An example from the MMLU dataset: ”Task699”. This question is from the biology domain, which requires domain knowledge at a high school level. MMLU: Task700 The dimerization of NO2(g) to N2O4(g) is an endothermic process. Which of the follo…
Figure 20
Figure 20. Figure 20: Example for Task700 (MMLU). An example from the MMLU dataset: ”Task700”. This question is from the chemistry domain, which requires domain knowledge at a high school level. 30 [PITH_FULL_IMAGE:figures/full_fig_p032_20.png]
Figure 21
Figure 21. Figure 21: Example for Task701 (MMLU). An example from the MMLU dataset: ”Task701”. This question is from the computer science domain, which requires domain knowledge at a high school level. MMLU: Task708 An object is released from rest and falls a distance h during the first se…
Figure 22
Figure 22. Figure 22: Example for Task708 (MMLU). An example from the MMLU dataset: ”Task708”. This question is from the physics domain, which requires domain knowledge at a high school level. 31 [PITH_FULL_IMAGE:figures/full_fig_p033_22.png]
Figure 23
Figure 23. Figure 23: Example for Task710 (MMLU). An example from the MMLU dataset: ”Task710”. This question is from the statistics domain, which requires domain knowledge at a high school level. MMLU: Task729 A seller sold his boat to a buyer. During negotiations, the buyer said that he p…
Figure 24
Figure 24. Figure 24: Example for Task729 (MMLU). An example from the MMLU dataset: ”Task729”. This question is from the law domain, which requires domain knowledge at a professional level. 32 [PITH_FULL_IMAGE:figures/full_fig_p034_24.png]
Figure 25
Figure 25. Figure 25: Example for Task730 (MMLU). An example from the MMLU dataset: ”Task730”. This question is from the medicine domain, which requires domain knowledge at a professional level. GPQA A large gene has dozens of exons, of which the central ones code for folded triple helical…
Figure 26
Figure 26. Figure 26: Example for GPQA. An example from the GPQA dataset. This question is from the biology domain, which requires domain knowledge at a PhD level. 33 [PITH_FULL_IMAGE:figures/full_fig_p035_26.png]
Figure 27
Figure 27. Figure 27: Example for GSM8K. An example from the GSM8K dataset. This question is from the mathe￾matics domain, which requires domain knowledge at a grade school level. 34 [PITH_FULL_IMAGE:figures/full_fig_p036_27.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 2 canonical work pages

  1. [1]

    Specificity in En- glish for Academic Purposes (EAP): A Corpus Analysis of Lexical Bundles in Academic Writ- ing

    Ang, L. and K. Tan (2018). “Specificity in En- glish for Academic Purposes (EAP): A Corpus Analysis of Lexical Bundles in Academic Writ- ing”. In: 24, pp. 82–94. doi: 10.17576/3L- 2018-2402-07. Bach, Stephen H. et al. (Mar. 2022). Prompt- Source: An Integrated Development Environ- ment and Repository for Natural Language Prompts. arXiv: 2202.01279 [cs]. B...

    arXiv 2018

  2. [5]

    Kaplan, Jared et al. (2020). Scaling Laws for Neural Language Models . arXiv: 2001.08361 [cs.LG]. url: https : / / arxiv . org / abs / 2001.08361. Kojima, Takeshi et al. (2023). Large Language Models are Zero-Shot Reasoners. arXiv:

    arXiv 2020

  3. [7]

    Direct Prefer- ence Optimization: Your Language Model is Secretly a Reward Model

    6, pp. 380–384. Ouyang, Long et al. (Mar. 4, 2022).Training Lan- guage Models to Follow Instructions with Hu- man Feedback. doi: 10.48550/arXiv.2203. 02155. arXiv: 2203.02155 [cs] . url: http: //arxiv.org/abs/2203.02155. preprint. Petrov, Slav, Dipanjan Das, and Ryan McDonald (2011). A Universal Part-of-Speech Tagset . arXiv: 1104 . 2086 [cs.CL]. url: htt...

    arXiv doi:10.48550/arxiv.2203.02155 2011

  4. [36]

    An Information- theoretic Approach to Prompt Engineering Without Ground Truth Labels

    Curran Associates, Inc., pp. 53728–53741. url: https : / / proceedings . neurips . cc / paper _ files / paper / 2023 / file / a85b405ed65c6477a4fe8302b5e06ce7 - Paper-Conference.pdf. Rawte, Vipula et al. (Sept. 2023). Exploring the Relationship between LLM Hallucinations and Prompt Linguistic Nuances: Readability, For- mality, and Concreteness. arXiv: 230...

    arXiv 2023

  5. [41]

    Prompt Learning with Structured Semantic Knowl- edge Makes Pre-Trained Language Mod- els Better

    issn: 2398-6352. doi: 10.1038/s41746-024-01029-4 . Wang, Yizhong et al. (Oct. 2022). Super- NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks. arXiv: 2204.07705 [cs]. White, Jules et al. (2023). A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv: 2302.11382 [cs.SE] . url: https://arxiv.org/abs/2302...

    arXiv 2023

  6. [494]

    org / 2023

    url: https : / / aclanthology . org / 2023 . findings - acl

    work page 2023

  7. [1128]

    Measuring Mas- sive Multitask Language Understanding

    Dubey, Abhimanyu et al. (2024). The Llama 3 Herd of Models. arXiv: 2407.21783 [cs.AI] . url: https://arxiv.org/abs/2407.21783. Fellbaum, Christiane (1998). WordNet: An elec- tronic lexical database. MIT press. Hendrycks, Dan et al. (2021). “Measuring Mas- sive Multitask Language Understanding”. In: Proceedings of the International Conference on Learning R...

    arXiv 2024

  8. [2005]

    A tutorial on kernel den- sity estimation and recent advances

    14165 [cs]. Chen, Banghao et al. (2024). Unleashing the po- tential of prompt engineering in Large Lan- guage Models: a comprehensive review. arXiv: 2310.14735 [cs.CL] . url: https://arxiv. org/abs/2310.14735. 21 Chen, Yen-Chi (2017). “A tutorial on kernel den- sity estimation and recent advances”. In: Bio- statistics & Epidemiology 1.1, pp. 161–187. doi:...

    arXiv 2024

Show all 10 references
  1. [2205]

    Pre-Train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

    11916 [cs.CL]. Leidinger, Alina, Robert van Rooij, and Ekate- rina Shutova (Nov. 2023). The Language of Prompting: What Linguistic Properties Make a Prompt Successful? arXiv: 2311 . 01967 [cs]. Liu, Pengfei et al. (Sept. 2023). “Pre-Train, Prompt, and Predict: A Systematic Sur...

    2024 arXiv

  2. [3281]

    issn: 2079-9292. doi: 10 . 3390 / electronics12153281. 23 A Figures Figure 11: Prompt Templates. Overview of all the prompt templates used for performing the question- answering and reasoning tasks (MMLU, GPQA and GSM8K), and the LLM-as-a-Judge experiment for the adjective spe...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.