What Makes a Good Natural Language Prompt?

Do Xuan Long; Duy Dinh; Kenji Kawaguchi; Min-Yen Kan; Nancy F. Chen; Ngoc-Hai Nguyen; Shafiq Joty

REVIEW 3 major objections 6 minor 16 references

What Makes a Good Natural Language Prompt?

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that natural-language prompt quality can be decomposed into 21 properties across six dimensions, and that evaluating prompts this way yields transferable design guidance and better reasoning models.

desk verdict The 21-property taxonomy and coverage audit are genuinely useful and citable; the empirical headline about 'single-property enhancements' is not supported by the experiments as presented. read the letter →

arxiv 2506.06950 v1 pith:SNKL2HIR submitted 2025-06-07 cs.CL

Do Xuan Long , Duy Dinh , Ngoc-Hai Nguyen , Kenji Kawaguchi , Nancy F. Chen , Shafiq Joty , Min-Yen Kan This is my paper

classification cs.CL

keywords promptengineeringevaluationproperty-centricframeworkmeta-analysisLLM-as-a-judgereasoningtasksinstructiontuningoptimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper attempts to replace outcome-only prompt evaluation with a property-based vocabulary. It surveys more than 150 prompting papers and blogs, distills 21 prompt properties in six dimensions (communication, cognition, instruction, logic and structure, hallucination awareness, and responsibility), and shows how support for these properties is uneven across tasks and models. It then scores 969 human-curated prompts on all 21 properties using an LLM judge calibrated on 50 human annotations, finding 17 strong correlations, and runs reasoning experiments on three models. The central empirical claim is that single-property enhancements often outperform multi-property combinations, and that instruction-tuning on property-enhanced prompts improves reasoning. If right, prompt engineering can shift from trial-and-error performance hacking to a shared, inspectable language of prompt quality.

What carries the argument

The central object is the 21-property taxonomy, organized into six dimensions: communication, cognition, instruction, logic and structure, hallucination, and responsibility. The load-bearing measurement device is an LLM judge that rates each property on a 1–10 scale using explicit-cue instructions and an incremental rubric, calibrated against 50 human-labelled prompts. The chain of argument depends on this judge producing reliable scores for all 969 collected prompts, on the correlation matrix computed from those scores, and on the subsequent prompting and fine-tuning experiments that test whether enhancing selected properties changes reasoning performance.

What would settle it

Re-score a random sample of at least 200 prompts from the same collection with multiple independent human annotators or with a second, differently calibrated judge, and check whether the 17 correlation coefficients at or above 0.7 reappear. Equivalently, run a causal intervention that enhances exactly one property while holding all others fixed and test whether the performance shift predicted by the correlation-based recommendations actually occurs; if the correlations collapse or the interventions fail to move performance as predicted, the framework's empirical claims are falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a good natural language prompt is best understood not by the score it produces but by a profile of 21 properties in six dimensions: communication (token quantity, manner, interaction, politeness), cognition (intrinsic, extraneous, and germane load), instruction (objectives, external tools, metacognition, demonstrations, rewards), logic and structure (structural and contextual logic), hallucination (awareness and the factuality–creativity balance), and responsibility (bias, safety, privacy, reliability, societal norms). The authors claim that scoring 969 human-curated prompts on this rubric with an LLM judge reveals 17 strong correlations among properties, that some properties are universally supported while others are task-specific, and that in their reasoning-task experiments a single-property enhancement is often the most effective change. They further claim that instruction-tuning on property-enhanced prompts, with politeness as the test case, produces better and more robust reasoning models than tuning on unmodified prompts.

Load-bearing premise

The load-bearing premise is that GPT-4o scores all 21 properties correctly on the 969 collected prompts, with calibration based on only 50 human-labelled examples; if the judge systematically mis-scores properties, the claimed correlations and the design recommendations derived from them would not generalize.

Editorial extensions

If this is right

Prompt evaluation can shift from outcome-only metrics to a shared property vocabulary, making prompt quality inspectable and comparable across studies.
Optimizing for directness, clarity, and conciseness is likely to improve token efficiency, logical coherence, and extraneous-load reduction, so these properties should be treated as a bundle.
Combining multiple positive prompt properties does not guarantee better performance; single-property enhancements are often the strongest, so prompt optimizers should avoid indiscriminately stacking properties.
Instruction-tuning on property-enhanced prompts, at least for politeness, can yield reasoning models that outperform models tuned on unmodified instructions.
The meta-analysis shows property support is uneven across models and tasks, identifying research gaps where future work can test whether a property transfers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: because the property correlations are observational, they may reflect the writing style of the curated prompt collections rather than causal relations; structural equation modelling or intervention studies would be needed to separate co-occurrence from influence.
Beyond the paper: if single-property enhancements beat multi-property combinations because added instructions push prompts out of an LLM's training distribution, then sparse property optimization should outperform dense prompt rewriting on a wider range of models and tasks.
Beyond the paper: the 21-property rubric could serve as a reward signal for automatic prompt optimization, letting an optimizer search for prompts that maximize a weighted property score rather than task accuracy alone.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The real contribution here is the framework: twenty-one prompt-quality properties organized into six dimensions, drawing on Grice, cognitive load theory, and instruction design. That synthesis is new and will be useful to anyone working on prompt evaluation or human-AI interaction. The Table 1 audit—mapping which properties have evidence across tasks and models—is also valuable; it gives the field a quick way to see where the support is thin. The correlation analysis on 969 prompts is a reasonable first step, and I credit them for trying to calibrate the LLM judge against human labels, even if only on 50 examples.

The soft spots are real but concentrated in the empirical sections. The headline claim that 'single-property enhancements often have the greatest impact' rests on Table 2: three models, four tasks, hand-crafted prompt additions, no repeated seeds, no confidence intervals, no significance tests. Many of the differences are a point or two, and the pattern is inconsistent—politeness helps Llama on some tasks, hurts o3-mini almost everywhere. That does not support 'often' as a general principle. The fine-tuning experiment is narrower still: one model, one property, one 2,500-example dataset. The paper's own limitations paragraph admits the prompts were 'in their simplest form' and not optimized, which is honest but undercuts the strength of the empirical conclusions.

The correlation analysis also has a load-bearing assumption: a single GPT-4o judge scoring all 969 prompts, with thresholds chosen at 0.7 and 5/10. That could be fine, but without a larger human-validation set or a second judge on the full corpus, the specific correlation values are shaky. I also noticed the code and data are promised at a placeholder link, not actually released.

None of this kills the paper. The taxonomy stands independent of the experiments, and the survey and Table 1 are worth having. The right fix is to temper the abstract and Section 6 claims, label the experiments as preliminary, and either add proper statistical treatment or move them to a 'case study' framing. I would send this to peer review, but with a request for major revision on the empirical parts. The framework itself deserves to be in the literature.

Referee Report

3 major / 6 minor

Summary. The paper proposes a property- and human-centric framework for evaluating natural language prompts, synthesizing 21 properties across six dimensions from a survey of over 150 prompting papers and blogs. It then examines how prior studies support each property across models and tasks, analyzes correlations among properties in a curated set of 969 prompts using an LLM judge, and reports prompting and fine-tuning experiments on reasoning tasks. The central claims are that the 21-property framework provides a shared vocabulary for prompt quality, that correlation analysis yields actionable design recommendations, and that single-property enhancements often outperform multi-property combinations, with property-enhanced instruction tuning yielding better reasoning models.

Significance. If the framework and empirical findings hold, the paper would provide a useful synthesis of a fragmented literature and a common vocabulary for prompt evaluation, shifting attention from outcome-only metrics to interpretable properties. The authors are to be credited for a broad survey, a clearly presented taxonomy grounded in cognitive load theory, Grice's maxims, and instructional design, for the publication of code and data, for validating their LLM judge against human annotations (albeit on a small sample), and for replicating the correlation analysis with a second judge model in Appendix D. The open questions enumerated in Sections 4 and 5 are genuinely useful research directions. However, the empirical load-bearing claims—especially the headline 'single-property enhancements often have the greatest impact'—rest on thin, unreplicated experiments, and the correlation analysis depends on a single LLM judge calibrated on only 50 of 969 prompts. The framework itself is defensible as a conceptual contribution, but the paper's practical recommendations require stronger empirical support.

major comments (3)

[Section 5, Figure 1] The correlation analysis is load-bearing for the design recommendations, but its reliability is not established. The authors validate their GPT-4o judge on only 50 of 969 prompts, and they report that the original judging format produced Cohen's kappa below 0.15 for 15/21 properties, with improvements only after iterative prompt engineering; however, they do not report the final kappa values, confidence intervals, or agreement on held-out prompts not used to develop the rubric. The cutoffs for 'strong' correlations (>=0.7) and for discarding low-average pairs (<5/10) are chosen without justification or sensitivity analysis. Given that the 17/210 strong correlations in Figure 1 drive the recommendations, the paper should either provide a more thorough validation of the judge on a larger human-labeled sample, report kappa per property with confidence intervals, or explicitly frame the correlations as exploratory and model-dependent.
[Section 6.1, Table 2] The abstract and conclusion claim that 'single-property enhancements often have the greatest impact,' but this claim rests entirely on Table 2, which reports single runs with no seeds, variance, confidence intervals, or significance tests. Several differences are small relative to expected sampling noise (e.g., Llama-3.1 GSM8K remains at 82.0 with germane load; Qwen ARC-C goes from 59.5 to 62.0 with metacognition), and the pattern is inconsistent across models and tasks: on o3-mini nearly all additions degrade accuracy relative to zero-shot CoT, while on Llama MMLU the triple combination (69.5) is the best condition, not a single property. The paper's own limitations section concedes that the property-enhancement prompts are 'in their simplest form, without optimization.' A single unreplicated experiment on three models and four tasks cannot establish 'often' as a general principle; please report multiple seeds, effect sizes with confidence intervals, and a clear criterion for what counts as an improvement.
[Section 6.2, Table 3] The fine-tuning experiment supporting the claim that instruction-tuning on property-enhanced prompts 'can result in better reasoning models' uses one model (Qwen-2.5-7B-It), one property (politeness via appending 'Please'), and one 2,500-example sample from Alpaca-GPT-4o, with no replication across seeds or data subsets. The result is also surprising: Table 3 shows that the model fine-tuned on polite data outperforms the model fine-tuned on non-polite data even when evaluated with non-polite prompts on several tasks, and this interaction is not analyzed. This experiment is a proof-of-concept, not a general finding; the conclusion should be tempered accordingly, or the experiment should be extended to additional properties, models, and seeds.

minor comments (6)

[Section 3, Hallucination dimension] The definition of 'Balancing factuality with creativity' contains an apparent typo ('prioritize creativity over creativity and vice versa'), and the paper states that no prompting method has been observed for this criterion; please clarify whether this property should remain in the taxonomy or be moved to the open questions.
[Abstract and Introduction] The phrase 'quantifies natural language prompts' is stronger than what is delivered: the framework produces ordinal LLM-judged scores for 21 properties, not a validated quantitative instrument. Please align the wording with the evidence presented.
[Table 1] The model logos in Table 1 are difficult to reproduce in text and make the table hard to read; a text-based legend or a supplementary CSV with the counts would improve usability.
[Limitations] The limitations section acknowledges that the survey may omit relevant papers and that the 'Responsibility' dimension is too broad; these caveats should be connected to the corresponding claims in Sections 4 and 5 rather than appearing as afterthoughts.
[Appendix B] The paper refers to '150+ papers' but does not give the exact number or explicit inclusion/exclusion criteria for the survey; please state the exact corpus size and make the selection protocol reproducible.
[Appendix D] The Gemini-2.0-flash correlation results show that two correlations drop from 0.7 to 0.6 relative to GPT-4o; this is a useful robustness check, but it should be mentioned in the main text as a qualification of the 'strong correlation' claims rather than only in the appendix.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the taxonomy, correlation analysis, and prompting/fine-tuning experiments are not forced by fitted parameters or definitional identity; only minor non-load-bearing self-citations appear.

full rationale

The paper's central derivation chain is a meta-analysis that proposes 21 prompt properties grounded in external theories (Grice, Cognitive Load Theory, Gagne) and prior work, not in the paper's own conclusions. Section 5's correlation analysis is observational: GPT-4o is calibrated on 50 human-annotated prompts, then used to score 969 prompts; the resulting recommendations are explicitly hedged by the paper's own open questions (Oq6-Oq8) about whether correlations are causal or mere co-occurrence. Section 6 uses independent accuracy metrics on MMLU, CommonsenseQA, ARC-Challenge, and GSM8K; no parameter is fitted to the headline claim that single-property enhancements often have the greatest impact, so that claim is not equivalent to its inputs by construction, even though it is statistically fragile. The only self-citations (e.g., Long et al. 2024b for reliability, Long et al. 2025b for objectives, Long et al. 2025a for format-following difficulties) are supporting evidence rather than load-bearing premises, and the same properties are also supported by external citations. The Limitations section acknowledges corpus scope and unoptimized prompt additions, which are validity risks, not circularity. Overall, the paper is self-contained against external benchmarks; the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The core of the paper is a definitional framework, so the main 'free parameters' are analytic choices (correlation thresholds, prompt templates) rather than fitted constants. The empirical load rests on three domain assumptions: GPT-4o judge reliability, representativeness of the 969-prompt corpus, and the accuracy of the manual paper-to-property coding. The framework itself is an invented conceptual entity, operationalized by the judging prompts in Appendix E.

free parameters (3)

Strong-correlation threshold = 0.7
Chosen by authors to define 'strong correlations'; determines the 17/210 strong-correlation claim in Section 5.
Low-average exclusion threshold = 5/10
Pairs with both property averages below 5 are excluded from correlation interpretation, a post-hoc analytic choice affecting the recommendation set in Section 5.
Prompt-enhancement phrase templates = Exact strings (e.g., 'Please', 'Reflect on your prior knowledge...')
The specific hand-crafted phrases used in Section 6 operationalize each property; different phrasings could change the observed effects.

assumptions (4)

domain assumption GPT-4o can reliably judge the 21 prompt properties with human-level accuracy when given the calibrated rubric.
The correlation analysis scores all 969 prompts with GPT-4o-2024-11-20, validated against only 50 human-labelled prompts (Section 5 Methods, Appendix Figure 2).
domain assumption The 969-prompt corpus is representative of 'high-quality natural language prompts'.
Prompts are drawn from prompt-engineering papers, curated collections, Alpaca, Natural Instructions, and LMSYS (Section 5), but there is no sampling frame or external validation of representativeness.
domain assumption The paper-to-property assignments in Table 1 are accurate and complete enough for the imbalanced-support conclusions.
Manual coding of 150+ papers into 21 properties is subjective and the survey scope is acknowledged as incomplete (Section 3, Limitations).
domain assumption The 21 properties are mutually exclusive and exhaustively cover prompt quality.
The authors state the properties are 'non-overlapping' (Section 3) but acknowledge partial overlap between hallucination awareness and token quantity; exhaustiveness is asserted, not proven.

invented entities (1)

21 prompt-quality properties across six dimensions (token quantity, manner, interaction, politeness, intrinsic/extraneous/germane load, objectives, external tools, metacognition, demos, rewards… independent evidence
purpose: Provide a property-centric, human-centric framework for evaluating and optimizing natural language prompts.
Each property is operationalized with explicit judging prompts (Appendix E) and inter-rater agreement is reported, giving external researchers a falsifiable handle to apply or challenge the framework.

how reviews work

0 comments

Cite this review

Pith. "Pith review of What Makes a Good Natural Language Prompt?." pith.science (2026). https://pith.science/paper/SNKL2HIR

@misc{pith2026250606950,
  author       = {Pith},
  title        = {Pith review of: What Makes a Good Natural Language Prompt?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SNKL2HIR}},
  note         = {Machine review of arXiv:2506.06950}
}

read the original abstract

As large language models (LLMs) have progressed towards more human-like and human--AI communications have become prevalent, prompting has emerged as a decisive component. However, there is limited conceptual consensus on what exactly quantifies natural language prompts. We attempt to address this question by conducting a meta-analysis surveying more than 150 prompting-related papers from leading NLP and AI conferences from 2022 to 2025 and blogs. We propose a property- and human-centric framework for evaluating prompt quality, encompassing 21 properties categorized into six dimensions. We then examine how existing studies assess their impact on LLMs, revealing their imbalanced support across models and tasks, and substantial research gaps. Further, we analyze correlations among properties in high-quality natural language prompts, deriving prompting recommendations. We then empirically explore multi-property prompt enhancements in reasoning tasks, observing that single-property enhancements often have the greatest impact. Finally, we discover that instruction-tuning on property-enhanced prompts can result in better reasoning models. Our findings establish a foundation for property-centric prompt evaluation and optimization, bridging the gaps between human--AI communication and opening new prompting research directions.

Figures

Figures reproduced from arXiv: 2506.06950 by the authors.

**Figure 2.** Agreements between human evaluators and LLM-based evaluation methods measured by Cohen’s Kappa. [PITH_FULL_IMAGE:figures/full_fig_p024_2.png] view at source ↗

**Figure 3.** Correlations of properties evaluated by gemini-2.0-flash. We do not consider correlations between pairs of properties [PITH_FULL_IMAGE:figures/full_fig_p034_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 4 canonical work pages

[2]

Journal of Machine Learning Research, 25(70):1–53

Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Karl Cobbe, Vineet Kosaraju...

arXiv 2018
[3]

In Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128, Miami, Florida, USA

A survey on in-context learning. In Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128, Miami, Florida, USA. Association for Computational Linguistics. Ehsan Doostmohammadi, Oskar Holmström, and Marco Kuhlmann. 2024. How reliable are automatic eval- uation methods for instruction-tuned LLMs? In Findin...

arXiv 2024
[5]

In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 6465–6488, Singapore

Enabling large language models to generate text with citations. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 6465–6488, Singapore. Associa- tion for Computational Linguistics. Zhibin Gou, Qingyan Guo, and Yujiu Yang. 2023. MvP: Multi-view prompting improves aspect sentiment tu- ple prediction. In Proce...

arXiv 2023
[6]

In Findings of the Associ- ation for Computational Linguistics: EMNLP 2024, pages 3830–3842, Miami, Florida, USA

Enhancing incremental summarization with structured representations. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2024, pages 3830–3842, Miami, Florida, USA. Association for Computational Linguistics. Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023. Towards mitigating LLM hallucination via self refl...

arXiv 2024
[7]

In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1658–1677, Bangkok, Thailand

LongLLMLingua: Accelerating and enhanc- ing LLMs in long context scenarios via prompt com- pression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1658–1677, Bangkok, Thailand. Association for Computational Linguistics. Zhiwei Jiang, Tianyi Gao, Yafeng Yin, Meng Liu, Hua Yu, Zif...

arXiv 2024
[8]

Association for Computational Linguistics

Do prompt positions really matter? In Find- ings of the Association for Computational Linguis- tics: NAACL 2024, pages 4102–4130, Mexico City, Mexico. Association for Computational Linguistics. Hugo Mercier and Dan Sperber. 2011. Why do hu- mans reason? arguments for an argumentative theory. Behavioral and Brain Sciences, 34(2):57–74. Grégoire Mialon, Rob...

work page 2024
[10]

arXiv preprint arXiv:2402.07927

A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927. Gregory Schraw and David Moshman. 1995. Metacog- nitive theories. Educational psychology review , 7:351–371. Shivam Shandilya, Menglin Xia, Supriyo Ghosh, Huiqiang Jiang, Jue Zhang, Qianhui Wu, and Victor Rühle. 2024. Taco-rl: ...

arXiv 1995
[11]

In Advances in Neural Information Processing Systems, volume 35, pages 14274–14289

Test-time prompt tuning for zero-shot gener- alization in vision-language models. In Advances in Neural Information Processing Systems, volume 35, pages 14274–14289. Chenglei Si, Dan Friedman, Nitish Joshi, Shi Feng, Danqi Chen, and He He. 2023a. Measuring induc- tive biases of in-context learning with underspeci- fied demonstrations. In Proceedings of th...

arXiv 2024

Show all 16 references

[12]

Stanford In- stitute for Human-Centered Artificial Intelligence (HAI)

Fine-tuning and prompt optimization: Two great steps that work better together. Stanford In- stitute for Human-Centered Artificial Intelligence (HAI). Sam Spilsbury, Pekka Marttinen, and Alexander Ilin
[13]

In Proceedings of the 2024 Conference on Em- pirical Methods in Natural Language Processing , pages 15960–15991, Miami, Florida, USA

Generating demonstrations for in-context com- positional generalization in grounded language learn- ing. In Proceedings of the 2024 Conference on Em- pirical Methods in Natural Language Processing , pages 15960–15991, Miami, Florida, USA. Associa- tion for Computational Lingui...

2024 arXiv
[14]

In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7669–7683, Toronto, Canada

Answering ambiguous questions via iterative prompting. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7669–7683, Toronto, Canada. Association for Computational Linguistics. Yueqing Sun, Yu Zhang, Le Q...

2022 arXiv
[15]

In Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 16230–16244, Miami, Florida, USA

Distract large language models for automatic jailbreak attack. In Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 16230–16244, Miami, Florida, USA. Association for Computational Linguistics. Binfeng Xu, Xukun Liu, Hua Shen, Zey...

2024 arXiv
[16]

Responsibility

Prompt refinement with image pivot for text- to-image generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 941–954, Bangkok, Thailand. Association for Computational Linguistics. Hanning Zhang, Shi...

2024 arXiv
[2022]

gradient descent

Training language models to follow instruc- tions with human feedback. In Advances in neural information processing systems, volume 35, pages 27730–27744. Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-...

2024
[2023]

Rupkatha Journal on Interdisciplinary Studies in Hu- manities, 15(4)

Breaking the bias: Gender fairness in llms using prompt engineering and in-context learning. Rupkatha Journal on Interdisciplinary Studies in Hu- manities, 15(4). Jessica Maria Echterhoff, Yao Liu, Abeer Alessa, Ju- lian McAuley, and Zexue He. 2024. Cognitive bias in decision-...

2024 arXiv
[2024]

Association for Computational Linguistics

Seemingly plausible distractors in multi-hop reasoning: Are large language models attentive read- ers? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2514–2528, Miami, Florida, USA. Association for Computational Linguistics. Te...

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [2]

Journal of Machine Learning Research, 25(70):1–53

Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Karl Cobbe, Vineet Kosaraju...

arXiv 2018

[2] [3]

In Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128, Miami, Florida, USA

A survey on in-context learning. In Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128, Miami, Florida, USA. Association for Computational Linguistics. Ehsan Doostmohammadi, Oskar Holmström, and Marco Kuhlmann. 2024. How reliable are automatic eval- uation methods for instruction-tuned LLMs? In Findin...

arXiv 2024

[3] [5]

In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 6465–6488, Singapore

Enabling large language models to generate text with citations. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 6465–6488, Singapore. Associa- tion for Computational Linguistics. Zhibin Gou, Qingyan Guo, and Yujiu Yang. 2023. MvP: Multi-view prompting improves aspect sentiment tu- ple prediction. In Proce...

arXiv 2023

[4] [6]

In Findings of the Associ- ation for Computational Linguistics: EMNLP 2024, pages 3830–3842, Miami, Florida, USA

Enhancing incremental summarization with structured representations. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2024, pages 3830–3842, Miami, Florida, USA. Association for Computational Linguistics. Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023. Towards mitigating LLM hallucination via self refl...

arXiv 2024

[5] [7]

In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1658–1677, Bangkok, Thailand

LongLLMLingua: Accelerating and enhanc- ing LLMs in long context scenarios via prompt com- pression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1658–1677, Bangkok, Thailand. Association for Computational Linguistics. Zhiwei Jiang, Tianyi Gao, Yafeng Yin, Meng Liu, Hua Yu, Zif...

arXiv 2024

[6] [8]

Association for Computational Linguistics

Do prompt positions really matter? In Find- ings of the Association for Computational Linguis- tics: NAACL 2024, pages 4102–4130, Mexico City, Mexico. Association for Computational Linguistics. Hugo Mercier and Dan Sperber. 2011. Why do hu- mans reason? arguments for an argumentative theory. Behavioral and Brain Sciences, 34(2):57–74. Grégoire Mialon, Rob...

work page 2024

[7] [10]

arXiv preprint arXiv:2402.07927

A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927. Gregory Schraw and David Moshman. 1995. Metacog- nitive theories. Educational psychology review , 7:351–371. Shivam Shandilya, Menglin Xia, Supriyo Ghosh, Huiqiang Jiang, Jue Zhang, Qianhui Wu, and Victor Rühle. 2024. Taco-rl: ...

arXiv 1995

[8] [11]

In Advances in Neural Information Processing Systems, volume 35, pages 14274–14289

Test-time prompt tuning for zero-shot gener- alization in vision-language models. In Advances in Neural Information Processing Systems, volume 35, pages 14274–14289. Chenglei Si, Dan Friedman, Nitish Joshi, Shi Feng, Danqi Chen, and He He. 2023a. Measuring induc- tive biases of in-context learning with underspeci- fied demonstrations. In Proceedings of th...

arXiv 2024