Pith. sign in

REVIEW 3 major objections 4 minor 26 cited by

Does Prompt Formatting Have Any Impact on LLM Performance?

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Prompt template choice alone causes statistically significant performance differences in GPT models, with no universally optimal format.

desk verdict Prompt formatting clearly moves GPT scores, but the paper's 'format vs. wording' story is confounded by the templates it prints, and the significance tests are post hoc. read the letter →

arxiv 2411.10541 v1 pith:4H54EWAQ submitted 2024-11-15 cs.CL cs.LG

Jia He , Mukund Rungta , David Koleczek , Arshdeep Sekhon , Franklin X Wang , Sadid Hasan This is my paper · ORCID
classification cs.CLcs.LG
keywords promptformattingtemplateslargelanguagemodelsGPT-3.5GPT-4benchmarkevaluationsensitivitycodegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the structural format of a prompt — plain text, Markdown, JSON, or YAML — changes how well GPT models perform, even when the words are held constant. Across six benchmarks and four GPT models, the authors find statistically significant gaps between the best and worst templates, with GPT-3.5-turbo varying by up to 40% in a code translation task and much larger relative swings on some benchmarks. They conclude that fixed-template evaluations can misrepresent model capability, and that no single format is universally best. The paper also reports that newer, larger GPT-4 models are less affected by format changes and produce more consistent answers across templates.

What carries the argument

The controlled template comparison is the load-bearing mechanism: the same persona, instructions, examples, output-format instructions, and user ask are repackaged into plain text, Markdown, JSON, and YAML, so structure is the only intended variable. Four measurements carry the argument: sensitivity (matched-pairs t-test between best and worst template), consistency (proportion of identical model answers between template pairs), transferability (Intersection-over-Union of the sets of top templates between model pairs), and stability (coefficient of mean deviation of scores across all templates). The t-test and the coefficient of mean deviation do the main work of showing that format effects are real and that newer models are less sensitive to them.

What would settle it

Build a pair of templates that are byte-for-byte identical except for pure markup characters—say, Markdown headers and dashes versus the same words with no markup—and measure the same tasks. If the performance gap collapses to near zero, the reported effect is caused by incidental wording changes, not by structural formatting; if the gap persists, the structural claim holds.

Watch

Extended reading notes

Core claim

The paper's central claim is that prompt markup alone is a significant, statistically detectable variable in GPT performance. Using a matched-pairs t-test on the best and worst templates for every model–benchmark pair, the authors find p-values below 0.05 in all but one case (GPT-4-1106-preview on HumanEval), demonstrating that template choice matters. They further claim there is no universally optimal format: GPT-3.5 prefers JSON, GPT-4 favors Markdown, and IoU overlap of top templates between model series is often below 0.2. Larger models are more consistent (above 0.5) and more stable, with GPT-4-1106-preview showing the lowest coefficient of mean deviation, below 0.036 across benchmarks.

Load-bearing premise

The paper assumes that the four templates are semantically identical, so any performance difference is caused by structure alone; but JSON and YAML rephrase the instructions as labeled fields, which could shift emphasis in a way that is more than purely syntactic.

Editorial extensions

If this is right

  • If template choice shifts scores by up to 40%, single-template benchmark results for GPT models are not stable estimates of capability; reporting a range over templates would be more informative.
  • Prompt format is not transferable across model families: a template tuned for GPT-3.5 may be suboptimal for GPT-4, so prompt engineering must be model-specific.
  • GPT-4-turbo's greater stability means format sensitivity is partly a model-capability issue, not a fixed property of all LLMs.
  • Future LLM evaluations should either report performance across a set of formats or explicitly justify the choice of one format, since that choice can change conclusions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the format effect likely generalizes beyond GPT to other instruction-tuned LLMs; if so, cross-paper comparisons that use different templates are not apples-to-apples even when the underlying model is similar.
  • Editorial inference: the JSON and YAML templates do not preserve wording exactly—they relabel instructions as named fields—so part of the 'format' effect may be an incidental wording effect; a pure-markup control would separate the two.
  • Editorial inference: a practical evaluation standard would be to run each model on a small portfolio of templates and report the mean and spread, making template sensitivity an explicit, cheap-to-report metric.
  • Editorial inference: the result suggests a testable prediction about training—models trained on heavily structured or JSON-heavy data might show smaller format effects on those formats, which could be checked with open-weight models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper empirically studies whether the choice of prompt template (plain text, Markdown, JSON, YAML) affects the performance of OpenAI GPT models (gpt-35-turbo-0613, gpt-35-turbo-16k-0613, gpt-4-32k-0613, gpt-4-1106-preview) on six benchmarks spanning natural language understanding, code generation, and code translation. The authors report statistically significant differences between the best- and worst-performing templates for most model/dataset combinations, no universally optimal format across models, higher answer consistency for larger models, and low transferability of the best template across model families. The central claim is that fixed prompt templates can misrepresent LLM capability because template structure alone drives performance differences.

Significance. If the central claim holds, the paper makes a useful contribution by extending prior work on fine-grained prompt sensitivity (e.g., Sclar et al., 2023; Voronov et al., 2024) to whole-template choices on a broad set of tasks and proprietary models. The study is systematic: it uses standard benchmarks, multiple GPT versions, and a reasonably large set of tasks, and it explicitly addresses sensitivity, consistency, and transferability as separate questions. The practical implication—that fixed-template evaluations should be interpreted with caution—is important for the LLM evaluation community. However, the paper's causal attribution to 'structure and syntax' is undermined by a confound in the template design (see Major Comment 1), and the headline significance testing is statistically under-powered relative to the post hoc selection of extremes. These issues do not necessarily invalidate the qualitative observation that template choices matter, but they require revision before the paper's central claim is supported.

major comments (3)
  1. [Section 2.2 and Appendix C (Table 2)] The statement that 'the content of each placeholder stays the same across different prompt formats. The only differences are in structure and syntax' is contradicted by the templates shown in Table 2. The Markdown, YAML, and JSON templates introduce semantic labels such as 'Persona', 'Instructions', 'Output Format', 'User question', and 'Task', which are entirely absent from the plaintext template (where the user input appears as a bare {INPUT} and the system content as an unlabeled paragraph). These labels are additional lexical tokens with semantic content, and they are likely to affect model behavior independently of the syntactic structure (e.g., telling the model the input is a 'Task' may activate different instruction-following behavior). Thus the observed performance differences cannot be cleanly attributed to 'structure and syntax' as the paper claims. This is load-bearing because the paper explicitly frames its novelty as studying 'global prompt format modifications' and concludes that format alone explains the differences. The authors should either redesign the templates so that the same words appear across all conditions (e.g., add the same labels to plaintext as plain text headings), or explicitly reframe the manipulation as 'template choice including structural and lexical formatting elements' and adjust the wording in Sections 2.2 and 6 accordingly.
  2. [Section 3.1 and Table 1] The significance analysis selects the single best and worst templates post hoc from the four formats on the same data and then applies a one-sided matched-pairs t-test to those two extremes. Under the null hypothesis, the maximum and minimum of four correlated performance metrics are expected to differ by chance, and no multiple-comparison correction is applied; the reported p-values therefore overstate the significance of the comparison. This issue does not by itself overturn the qualitative conclusion—many of the raw differences are large—but the statistical evidence as presented is not valid. The authors should use a permutation test on the range across all four templates, apply a proper multiple-comparison correction, or report pairwise comparisons for all format pairs. Additionally, the abstract's claim of 'up to 40%' variation in a code translation task is not supported by Table 1: the CODEXGLUE differences are roughly 10–12 BLEU points (relative ~18%), and the HumanEval differences are on accuracy/pass@1, not translation. The paper should identify the exact contrast that yields the 40% figure.
  3. [Section 5.1 and Figure 3] The IoU transferability metric depends on the definition of 'top-performing templates' as those with 'statistically indistinguishable performance' from the best template, determined by a matched-pairs t-test. With only four templates, the resulting sets are highly sensitive to the significance threshold and inherit the multiple-comparison problem noted above. The paper does not report which templates are included in the top set for each model, so the IoU values in Figure 3 cannot be independently interpreted. Since transferability is one of the three main research questions, the authors should either provide the full set definitions and a more robust statistical procedure, or clearly report the raw pairwise differences that support the low-transferability conclusion.
minor comments (4)
  1. [Abstract] The abstract contains a typo: 'Y AML' should be 'YAML'.
  2. [Figure 4 caption] The caption says 'pass@1 to hecks if the the generated code passes' — this should be 'pass@1 checks if the generated code passes'.
  3. [Appendix E.1] The heading 'Additional results on model performance under all templates across benchamrks' has a typo: 'benchamrks' should be 'benchmarks'.
  4. [Section 2.3 / Appendix B] The paper does not state the sampling temperature used for the tasks other than MMLU and FIND (which are explicitly temperature-zero). Since consistency is measured on MMLU and FIND only, it is important to know whether the sensitivity results for the other benchmarks were obtained with temperature zero or with sampling, as this affects the interpretation of score variability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical measurement study whose conclusions rest on direct benchmark observations, not on premises that encode those conclusions.

full rationale

This paper makes no derived claim that reduces, by construction or by self-citation, to its own inputs. It measures GPT model performance on six external benchmarks under four prompt templates and reports observed score differences, t-test p-values, consistency proportions, and IoU overlaps. The central claim—that template choice is associated with statistically significant performance variation—is an empirical finding from the measured scores, not a consequence of any definition or fitted parameter. No parameter is fit to a target result and then renamed as a prediction; no uniqueness theorem or load-bearing prior result is imported from the authors' own work. The Section 2.2 assertion that 'the content of each placeholder stays the same across different prompt formats. The only differences are in structure and syntax' is a manipulation-check assumption. Appendix C shows that the JSON and YAML templates add semantic labels such as 'Persona', 'Instructions', and 'Task' that are absent from the plaintext template, which means the study may confound format syntax with incidental wording. That is a threat to construct validity or causal attribution, but it is not circularity: the paper's conclusions about performance variation across templates would still be supported by the measured differences, and the conclusion is not derived from the same-content assumption. The reader's skeptical attack is best classified as a correctness/validity concern, not as a circular derivation. Accordingly, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on an experimental comparison that assumes semantic equivalence of templates, valid statistical testing under post hoc selection, and API output stability. These are unverified assumptions rather than fitted parameters, and the paper does not introduce new entities.

assumptions (3)
  • domain assumption The four prompt templates (plain text, Markdown, YAML, JSON) are semantically equivalent, differing only in structure and syntax.
    Section 2.2 states that content is kept the same and only structure varies, but this equivalence is asserted, not measured; structured fields may change emphasis.
  • domain assumption Matched-pairs t-tests comparing the best and worst performing formats are valid despite the post hoc selection of those formats.
    Section 3.1 selects max and min formats from the same data; the test assumes this selection does not inflate significance, but no multiple-comparison correction is applied.
  • domain assumption GPT API outputs are stable enough (temperature and nondeterminism) that across-format differences are not due to sampling noise.
    The paper does not report temperature for all tasks or repeated runs; consistency was only measured on MMLU and FIND.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Does Prompt Formatting Have Any Impact on LLM Performance?." pith.science (2026). https://pith.science/paper/4H54EWAQ

@misc{pith2026241110541,
  author       = {Pith},
  title        = {Pith review of: Does Prompt Formatting Have Any Impact on LLM Performance?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4H54EWAQ}},
  note         = {Machine review of arXiv:2411.10541}
}
read the original abstract

In the realm of Large Language Models (LLMs), prompt optimization is crucial for model performance. Although previous research has explored aspects like rephrasing prompt contexts, using various prompting techniques (like in-context learning and chain-of-thought), and ordering few-shot examples, our understanding of LLM sensitivity to prompt templates remains limited. Therefore, this paper examines the impact of different prompt templates on LLM performance. We formatted the same contexts into various human-readable templates, including plain text, Markdown, JSON, and YAML, and evaluated their impact across tasks like natural language reasoning, code generation, and translation using OpenAI's GPT models. Experiments show that GPT-3.5-turbo's performance varies by up to 40\% in a code translation task depending on the prompt template, while larger models like GPT-4 are more robust to these variations. Our analysis highlights the need to reconsider the use of fixed prompt templates, as different formats can significantly affect model performance.

Figures

Figures reproduced from arXiv: 2411.10541 by the authors.

Figure 1
Figure 1. An example to demonstrate how prompt for [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Intersection over Union (IoU) scores for top [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Model performance across prompt formats on MMLU, HumanEval and CODEXGLUE. Performance [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Performance spread across models on MMLU benchmark per domain. Wide performance spread is [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Coefficient of mean deviation (CMD) of scalar [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Heatmap of IoU values for other benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Performance of Consistency for FIND dataset [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Dotplot of model performance across prompt formats on all benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 26 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Unable to Forget: Proactive Interference Reveals Working Memory Limits in LLMs Beyond Context Length

    cs.CL 2025-06 conditional novelty 7.0 of 10

    LLM retrieval accuracy decays log-linearly with the number of interfering prior updates, and 'forget' instructions barely improve it.

  2. Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Across five LLMs, prompt format effects are real but model- and scale-specific; instruction-following collapses by 80 rules, and context pressure causes refusal, not hallucination.

  3. Which Algorithm Specification Formats Help Language Models Implement Machine Learning Algorithms?

    cs.SE 2026-07 conditional novelty 6.0 of 10

    LaTeX-style pseudocode helps most under sparse algorithm specs, but with complete details format effects shrink or vanish by model, so authors should write implementation rules explicitly instead of relying on surface format.

  4. Compression, structure, and executor capability: a controlled real-cost decomposition of language-model agent skill optimisation

    cs.SE 2026-07 accept novelty 6.0 of 10

    On 40 software-engineering agent tasks, executor model tier dominates skill optimisations: no shortening, structure, scoped loading, or compiler tier beats raw skills on quality or real cost.

  5. CARE-MH: Towards Unified, Reproducible, and Comparable Evaluation of Mental Health LLMs

    cs.HC 2026-05 conditional novelty 6.0 of 10

    CARE-MH is a unified evaluation framework showing that mental-health LLM benchmark results are strongly affected by evaluator-model stability and metric definitions.

  6. Auditing LLM Editorial Bias in News Media Exposure

    cs.CY 2025-10 conditional novelty 6.0 of 10

    Compared with Google News, GPT-4o-Mini, Claude-3.7-Sonnet, and Gemini-2.0-Flash surface fewer unique news outlets, distribute attention more unevenly, and lean ideologically in system-specific ways.

  7. Scaling Truth: The Confidence Paradox in AI Fact-Checking

    cs.SI 2025-09 conditional novelty 6.0 of 10

    Across LLM fact-checking, model scale correlates with an inverse pattern of accuracy and decisiveness: smaller models are overconfident and less accurate, larger models are accurate but overly cautious.

  8. From Canonical to Complex: Benchmarking LLM Capabilities in Undergraduate Thermodynamics

    physics.ed-ph 2025-08 conditional novelty 6.0 of 10

    On a new 50-item thermodynamics benchmark, the best LLM scored 82%, below the authors' 95% tutoring-safety threshold, with diagram-based questions near chance.

  9. Prompt Orchestration Markup Language

    cs.HC 2025-08 conditional novelty 6.0 of 10

    POML is a markup language that structures LLM prompts, embeds multimodal data, and decouples formatting via stylesheets, with case studies showing strong prompt format sensitivity.

  10. PDLogger: Automated Logging Framework for Practical Software Development

    cs.SE 2025-07 conditional novelty 6.0 of 10

    PDLogger generates complete multi-log statements for Java methods by combining block-aware LLM prompts, backward slicing, and refinement, outperforming prior single-log tools.

  11. Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning

    cs.AI 2025-07 conditional novelty 6.0 of 10

    Training a Qwen2.5-7B model with GRPO on math and puzzle data improves both domains, code transfer depends on the starting model, and template or reward mismatches sharply hurt performance.

  12. Existing Large Language Model Unlearning Evaluations Are Inconclusive

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Existing LLM unlearning evaluations are inconclusive: they can inject new information, depend heavily on task format, and rely on spurious correlations.

  13. Large Language Model Chatbot Conversations vs Public Health Materials and Parental HPV Vaccination Intentions: A Randomized Clinical Trial

    cs.CY 2025-04 conditional novelty 6.0 of 10

    In 930 vaccine-hesitant parents, GPT-4o chatbot conversations increased immediate HPV vaccination intent by 7 to 10 points but not beyond public health materials, and effects vanished by 15 days.

  14. Tuning LLM Judge Design Decisions for 1/1000 of the Cost

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A multi-fidelity, multi-objective search finds cheap open-weight LLM judges that match or outperform prior judge designs on several benchmarks.

  15. A Knowledge-Injection Framework for Zero-Shot Adaptation of LLMs to Delirium Prediction

    cs.CL 2026-05 conditional novelty 5.0 of 10

    Inference-time injection of a GPT-generated clinical risk framework improved zero-shot ICU delirium prediction for LLaMA 8B by 8.6 AUROC points, but the gain depended on report structure and did not appear with a more...

  16. Psychological Imagination Networks Show Cross-Population Centrality and Clustering Alignment in Humans That Large Language Models Fail to Replicate

    cs.AI 2025-10 conditional novelty 5.0 of 10

    Human vividness-rating networks are correlated across populations and cluster by questionnaire context, whereas LLM-derived networks are mostly degenerate single-clusters, showing a human-LLM divergence in imagined-sc...

  17. Decoupling Task-Solving and Output Formatting in LLM Generation

    cs.CL 2025-10 conditional novelty 5.0 of 10

    A decoding-time method that keeps the format in a separate module improves LLM accuracy by 1–6% with guaranteed format compliance on math, judging, and extraction.

  18. AllSummedUp: un framework open-source pour comparer les metriques d'evaluation de resume

    cs.CL 2025-08 conditional novelty 5.0 of 10

    On SummEval, LLM-based summary evaluators are expensive and unstable, and several published correlations did not reproduce when using open-weight models.

  19. LLM-Based Community Surveys for Operational Decision Making in Interconnected Utility Infrastructures

    cs.SI 2025-07 conditional novelty 5.0 of 10

    Simulated LLM personas can rank disaster repair priorities, and partial preference data recovers most of the full ranking.

  20. Format-Adapter: Improving Reasoning Capability of LLMs by Adapting Suitable Format

    cs.CL 2025-06 conditional novelty 5.0 of 10

    FORMAT-ADAPTER automatically generates and selects per-question reasoning formats for LLMs, improving vote-based accuracy by around 4.3% over prior multi-format methods.

  21. Improving Dialogue State Tracking through Combinatorial Search for In-Context Examples

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A combinatorial scoring method for retriever training data improves few-shot dialogue state tracking by 20x in data efficiency and by 12% in oracle upper-bound JGA over prior methods.

  22. ClassComet: Exploring and Designing AI-generated Danmaku in Educational Videos to Enhance Online Learning

    cs.HC 2025-04 conditional novelty 5.0 of 10

    AI-written danmaku, combining content and emotion types, can match human comment quality and significantly boost learner engagement and quiz gains in short educational videos.

  23. Joint Localization and Activation Editing for Low-Resource Fine-Tuning

    cs.CL 2025-02 conditional novelty 5.0 of 10

    JoLA jointly learns sparse attention-head selection and additive/multiplicative activation edits, outperforming LoRA and prior activation-editing baselines in low-resource LLM fine-tuning.

  24. AI as Equalizer or Amplifier? Task Complexity as the Moderating Factor for Human Expertise in Hybrid Intelligence Systems

    cs.HC 2025-10 conditional novelty 4.0 of 10

    Generative AI is a cognitive amplifier: output quality tracks user domain expertise, equalizing expert–novice performance on routine tasks but widening the gap on complex ones.

  25. Relative Bias: A Comparative Framework for Quantifying Bias in LLMs

    cs.CL 2025-05 conditional novelty 4.0 of 10

    A model is 'relatively biased' when its responses deviate from the consensus of a baseline LLM set, and this deviation can be scored by embedding distances or LLM judges plus equivalence tests.

  26. On the Surprising Efficacy of LLMs for Penetration-Testing

    cs.CR 2025-07 conditional novelty 3.0 of 10

    A critical review arguing that LLMs are surprisingly effective for penetration testing because the task is largely pattern-matching, while noting serious reliability, safety, and cost barriers to autonomous use.

Reference graph

Works this paper leans on

36 extracted references · 6 canonical work pages · cited by 26 Pith papers

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    June 2023

    Armen Aghajanyan. June 2023. https://twitter.com/ArmenAgha/status/1669084129261162497. Tweet: Susan and i found mmlu performance jump 6-10 points in the 40s by formatting multiple choice as (a) not a in mmlu (for internal model). all evaluation of llm’s are broken. evaluating a task requires marginalizing across all prompts that describe the task, not poi...

    arXiv 2023

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  4. [4]

    S \'e bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arxiv. arXiv preprint arXiv:2303.12712

    arXiv 2023

  5. [5]

    Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, et al. 2024. Stealing part of a production language model. arXiv preprint arXiv:2403.06634

    arXiv 2024

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  7. [7]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradb...

    arXiv 2022

  8. [8]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    arXiv 2020

Show all 36 references
  1. [9]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. https://arxiv.org/abs/2005.11401 Retrieval-augmented generation for knowledge-int...

    2021 arXiv

  2. [10]

    Yuanzhi Li, S \'e bastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463

    2023 arXiv

  3. [11]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. https://arxiv.org/abs/2307.03172 Lost in the middle: How language models use long contexts . Preprint, arXiv:2307.03172

    2023 arXiv

  4. [12]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664

    2021 arXiv

  5. [13]

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. https://arxiv.org/abs/2104.08786 Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity . Preprint, arXiv:2104.08786

    2022 arXiv

  6. [14]

    guidance

    Microsoft. guidance. https://github.com/guidance-ai

  7. [15]

    Microsoft. 2024. Azure openai service models. https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#gpt-4-and-gpt-4-turbo-preview. Accessed: 2024-03-26

    2024

  8. [16]

    OpenAI. 2023. Evals. https://github.com/openai/evals

    2023

  9. [17]

    OpenAI. 2024. https://openai.com/blog/new-embedding-models-and-api-updates New embedding models and api updates . Accessed: 2024-03-26

    2024

  10. [18]

    November 2023

    OpenAI. November 2023. https://openai.com/blog/new-models-and-developer-products-announced-at-devday Improved instruction following and json mode

    2023

  11. [19]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  12. [20]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  13. [21]

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. https://arxiv.org/abs/2402.07927 A systematic survey of prompt engineering in large language models: Techniques and applications . Preprint, arXiv:2402.07927

    2024 arXiv

  14. [22]

    Sarah Schwettmann, Tamar Rott Shaham, Joanna Materzynska, Neil Chowdhury, Shuang Li, Jacob Andreas, David Bau, and Antonio Torralba. 2023. https://arxiv.org/abs/2309.03886 Find: A function description benchmark for evaluating interpretability methods . Preprint, arXiv:2309.03886

    2023 arXiv

  15. [23]

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324

    2023 arXiv

  16. [24]

    Bangzhao Shu, Lechen Zhang, Minje Choi, Lavinia Dunagan, Dallas Card, and David Jurgens. 2023. You don't need a personality test to know these models are unreliable: Assessing the reliability of large language models on psychometric instruments. arXiv preprint arXiv:2311.09718

    2023 arXiv

  17. [25]

    Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. Table meets llm: Can large language models understand structured table data? a benchmark and empirical study. In The 17th ACM International Conference on Web Search and Data Mining (WSDM '24)

    2024

  18. [26]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  19. [27]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  20. [28]

    Anton Voronov, Lena Wolf, and Max Ryabinin. 2024. Mind your format: Towards consistent evaluation of in-context learning improvements. arXiv preprint arXiv:2401.06766

    2024 arXiv

  21. [29]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://arxiv.org/abs/2203.11171 Self-consistency improves chain of thought reasoning in language models . Preprint, arXiv:2203.11171

    2023 arXiv

  22. [30]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. https://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models . Preprint, arXiv:2201.11903

    2023 arXiv

  23. [31]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. https://arxiv.org/abs/2210.03629 React: Synergizing reasoning and acting in language models . Preprint, arXiv:2210.03629

    2023 arXiv

  24. [32]

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. https://arxiv.org/abs/2210.03493 Automatic chain of thought prompting in large language models . Preprint, arXiv:2210.03493

    2022 arXiv

  25. [33]

    Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh

    Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. https://arxiv.org/abs/2102.09690 Calibrate before use: Improving few-shot performance of language models . Preprint, arXiv:2102.09690

    2021 arXiv

  26. [34]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023. https://arxiv.org/abs/2303.17568 Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-...

    2023 arXiv

  27. [35]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  28. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.