{"id":"9fabbcb3-f7dd-4664-ae64-515214b479be","arxiv_id":"2411.10541","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Prompt format alone can shift GPT model scores by double-digit percentage points, with GPT-3.5 more affected than GPT-4 and no universally best template.","lead":"This study tests whether formatting the same prompt as plain text, Markdown, JSON, or YAML changes GPT model performance on reasoning, code, and translation tasks. It finds that format choice can produce large score swings, especially for GPT-3.5, and that no single format works best across models or tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'same content' claim in Section 2.2 is contradicted by the Appendix C templates, which add semantic labels like 'Persona', 'Instructions', and 'Task' to JSON/YAML but not to plaintext, confounding format with wording.","rationale":"The reader's weakest_assumption precisely identifies the semantic-equivalence confound, and I agree. Section 2.2's claim is contradicted by Appendix C templates, which introduce tokens like 'Persona', 'Instructions', 'OutputFormat', and 'Task' absent from the plaintext condition. This is more load-bearing than the statistical issues (post hoc selection, multiple comparisons) because the raw effect sizes are huge and would likely survive typical corrections; the core question is whether the effect is causally attributable to syntax/structure at all. If the added labels drive the effect, the paper's claim 'format matters' collapses into 'label wording matters', which is a far weaker and less novel conclusion. I also note that the abstract's 'up to 40%' in code translation is not supported by Tables 7 and 8 (max relative differences are roughly 11-18%), and the 'over 300%' claim on HumanEval is actually about 247% for GPT-4-32k (76.22/21.95); these overstatements erode confidence but do not themselves overturn the existence of format sensitivity. The proposed labeled-plaintext control would settle the attribution. Therefore the verdict remains CONDITIONAL: the paper should be accepted only if it demonstrates that the effect is not explained by the lexical labels, or if it re-frames the claim to include these formatting decisions.","tokens_in":13549,"tokens_out":8632,"duration_ms":78650,"concrete_test":"Construct a fifth 'labeled plaintext' condition that adds the same semantic labels to the original plaintext template without any markup syntax—for example, start the system message with 'Persona: You are...' and 'Instructions: ...', precede the example with 'Example:', and precede the user input with 'Task:'. Run the full benchmark suite on GPT-3.5-turbo-0613 (or at least MMLU and HumanEval). If the labeled plaintext produces accuracy close to JSON/YAML and far from the original plaintext, then the reported formatting effects are driven by label wording, not by structure/syntax. If it matches the original plaintext, the structural syntax itself is the cause. This directly tests the Section 2.2 equivalence assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central causal claim—that template syntax alone drives performance differences—rests on Section 2.2's assertion that 'the content of each placeholder stays the same across different prompt formats. The only differences are in structure and syntax.' Appendix C shows this is false. The plaintext template presents the user request as just '{INPUT}' and the system text as a prose paragraph without headings. The JSON template introduces the keys 'Persona', 'Instructions', 'OutputFormat', 'Example', and 'Task'; the YAML template introduces 'Persona:', 'Instructions:', 'Output_Format:', 'Examples:', and 'Task:'. These labels are additional tokens with semantic content not present in the plaintext condition. Telling the model the input is 'Task' may activate different instruction-following behavior than presenting it with no label. Similarly, the Markdown template adds headings like '## Output Format' and '### DOCUMENT'. The observed 200-300% swings could therefore be caused by the presence or position of these lexical labels rather than by the syntactic structure (Markdown headers, JSON braces, YAML indentation). If so, the paper's conclusion that fixed-template benchmarks can misrepresent capability survives, but the attribution to 'format' as opposed to 'wording' is unsupported. This is load-bearing because the paper's novelty claim is about global format changes; a confounded manipulation cannot support that.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper empirically studies whether the choice of prompt template (plain text, Markdown, JSON, YAML) affects the performance of OpenAI GPT models (gpt-35-turbo-0613, gpt-35-turbo-16k-0613, gpt-4-32k-0613, gpt-4-1106-preview) on six benchmarks spanning natural language understanding, code generation, and code translation. The authors report statistically significant differences between the best- and worst-performing templates for most model/dataset combinations, no universally optimal format across models, higher answer consistency for larger models, and low transferability of the best template across model families. The central claim is that fixed prompt templates can misrepresent LLM capability because template structure alone drives performance differences.","tokens_in":13773,"tokens_out":3679,"duration_ms":38151,"significance":"If the central claim holds, the paper makes a useful contribution by extending prior work on fine-grained prompt sensitivity (e.g., Sclar et al., 2023; Voronov et al., 2024) to whole-template choices on a broad set of tasks and proprietary models. The study is systematic: it uses standard benchmarks, multiple GPT versions, and a reasonably large set of tasks, and it explicitly addresses sensitivity, consistency, and transferability as separate questions. The practical implication—that fixed-template evaluations should be interpreted with caution—is important for the LLM evaluation community. However, the paper's causal attribution to 'structure and syntax' is undermined by a confound in the template design (see Major Comment 1), and the headline significance testing is statistically under-powered relative to the post hoc selection of extremes. These issues do not necessarily invalidate the qualitative observation that template choices matter, but they require revision before the paper's central claim is supported.","major_comments":[{"comment":"The statement that 'the content of each placeholder stays the same across different prompt formats. The only differences are in structure and syntax' is contradicted by the templates shown in Table 2. The Markdown, YAML, and JSON templates introduce semantic labels such as 'Persona', 'Instructions', 'Output Format', 'User question', and 'Task', which are entirely absent from the plaintext template (where the user input appears as a bare {INPUT} and the system content as an unlabeled paragraph). These labels are additional lexical tokens with semantic content, and they are likely to affect model behavior independently of the syntactic structure (e.g., telling the model the input is a 'Task' may activate different instruction-following behavior). Thus the observed performance differences cannot be cleanly attributed to 'structure and syntax' as the paper claims. This is load-bearing because the paper explicitly frames its novelty as studying 'global prompt format modifications' and concludes that format alone explains the differences. The authors should either redesign the templates so that the same words appear across all conditions (e.g., add the same labels to plaintext as plain text headings), or explicitly reframe the manipulation as 'template choice including structural and lexical formatting elements' and adjust the wording in Sections 2.2 and 6 accordingly.","section":"Section 2.2 and Appendix C (Table 2)"},{"comment":"The significance analysis selects the single best and worst templates post hoc from the four formats on the same data and then applies a one-sided matched-pairs t-test to those two extremes. Under the null hypothesis, the maximum and minimum of four correlated performance metrics are expected to differ by chance, and no multiple-comparison correction is applied; the reported p-values therefore overstate the significance of the comparison. This issue does not by itself overturn the qualitative conclusion—many of the raw differences are large—but the statistical evidence as presented is not valid. The authors should use a permutation test on the range across all four templates, apply a proper multiple-comparison correction, or report pairwise comparisons for all format pairs. Additionally, the abstract's claim of 'up to 40%' variation in a code translation task is not supported by Table 1: the CODEXGLUE differences are roughly 10–12 BLEU points (relative ~18%), and the HumanEval differences are on accuracy/pass@1, not translation. The paper should identify the exact contrast that yields the 40% figure.","section":"Section 3.1 and Table 1"},{"comment":"The IoU transferability metric depends on the definition of 'top-performing templates' as those with 'statistically indistinguishable performance' from the best template, determined by a matched-pairs t-test. With only four templates, the resulting sets are highly sensitive to the significance threshold and inherit the multiple-comparison problem noted above. The paper does not report which templates are included in the top set for each model, so the IoU values in Figure 3 cannot be independently interpreted. Since transferability is one of the three main research questions, the authors should either provide the full set definitions and a more robust statistical procedure, or clearly report the raw pairwise differences that support the low-transferability conclusion.","section":"Section 5.1 and Figure 3"}],"minor_comments":[{"comment":"The abstract contains a typo: 'Y AML' should be 'YAML'.","section":"Abstract"},{"comment":"The caption says 'pass@1 to hecks if the the generated code passes' — this should be 'pass@1 checks if the generated code passes'.","section":"Figure 4 caption"},{"comment":"The heading 'Additional results on model performance under all templates across benchamrks' has a typo: 'benchamrks' should be 'benchmarks'.","section":"Appendix E.1"},{"comment":"The paper does not state the sampling temperature used for the tasks other than MMLU and FIND (which are explicitly temperature-zero). Since consistency is measured on MMLU and FIND only, it is important to know whether the sensitivity results for the other benchmarks were obtained with temperature zero or with sampling, as this affects the interpretation of score variability.","section":"Section 2.3 / Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the NLP evaluation community, but the confound in the template design and the post hoc significance testing need to be addressed before the central claim can be accepted. The empirical phenomena—large score differences across templates and reduced sensitivity for GPT-4—are probably real, but the manuscript currently overstates the causal role of syntax. The authors should be encouraged to rerun or reframe the experiments; the revision is feasible within the paper's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core observation—that swapping a prompt between plain text, Markdown, YAML, and JSON changes GPT scores by large margins—is directionally right and worth knowing. What is genuinely new is the sweep: four GPT versions, six benchmarks, and a consistent sign that the GPT-4-1106-preview is more robust than its predecessors. The consistency metric and the IoU-based transferability analysis are reasonable ways to quantify the effect. I also credit the authors for including a limitations section and for reporting per-template numbers rather than just aggregates.\n\nThe soft spots are real but not fatal. The stress-test note is correct: the Appendix C templates are not 'the same content with only structure and syntax.' The plaintext condition is a bare paragraph with no section headers, while Markdown introduces '## Persona', '## Instructions', '## Output Format', YAML introduces 'Persona:', 'Instructions:', 'Task:', and JSON wraps everything in named keys. Those labels are additional tokens with semantic content, so the manipulation changes wording, not just syntactic scaffolding. That doesn't overturn the finding that scores vary—it just means the paper's central novelty claim, that global structural format per se drives the effect, is unsupported as stated. The authors would need a control where every condition carries the same labels, or a framing that treats 'format' as including these labels.\n\nThe statistics also deserve a harder look. Table 1 identifies the best and worst formats post hoc and then runs one-sided matched-pair t-tests on that same data. With only four formats and no multiple-comparison correction, the p-values overstate the evidence. The abstract's 'up to 40%' is also on a code translation task, but the body contains swings much larger in relative terms (GPT-4-32k on HumanEval jumping from 21.95 to 76.22), and there's a model-name typo ('1104' vs '1106') plus no released code or prompts. These are inconsistencies and omissions, not fatal flaws.\n\nOverall, the paper is a useful empirical map, and the direction of the headline claim survives the confound: fixed-template benchmarks can misrepresent capability, especially for smaller GPT-3.5 models. The attribution to pure syntax is what needs repair. This deserves a serious referee, not a desk reject, with a request for template controls, corrected statistics, and artifact release. I'd send it to review, but I would not cite it as a clean result until the confound is addressed.","headline":"Prompt formatting clearly moves GPT scores, but the paper's 'format vs. wording' story is confounded by the templates it prints, and the significance tests are post hoc.","tokens_in":14320,"tokens_out":1886,"would_cite":false,"duration_ms":19995,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompt template choice alone causes statistically significant performance differences in GPT models, with no universally optimal format.","keywords":["prompt formatting","prompt templates","large language models","GPT-3.5","GPT-4","benchmark evaluation","prompt sensitivity","code generation"],"falsifier":"Build a pair of templates that are byte-for-byte identical except for pure markup characters—say, Markdown headers and dashes versus the same words with no markup—and measure the same tasks. If the performance gap collapses to near zero, the reported effect is caused by incidental wording changes, not by structural formatting; if the gap persists, the structural claim holds.","tokens_in":13353,"feed_emoji":"🧩","tokens_out":7565,"duration_ms":59120,"temperature":0.7,"pith_summary":"This paper tries to establish that the structural format of a prompt — plain text, Markdown, JSON, or YAML — changes how well GPT models perform, even when the words are held constant. Across six benchmarks and four GPT models, the authors find statistically significant gaps between the best and worst templates, with GPT-3.5-turbo varying by up to 40% in a code translation task and much larger relative swings on some benchmarks. They conclude that fixed-template evaluations can misrepresent model capability, and that no single format is universally best. The paper also reports that newer, larger GPT-4 models are less affected by format changes and produce more consistent answers across templates.","feed_headline":"Prompt template alone swings GPT-3.5 scores by up to 40%","feed_subtitle":"Four wrappers of identical text produce different results on six benchmarks, and no format wins everywhere.","key_machinery":"The controlled template comparison is the load-bearing mechanism: the same persona, instructions, examples, output-format instructions, and user ask are repackaged into plain text, Markdown, JSON, and YAML, so structure is the only intended variable. Four measurements carry the argument: sensitivity (matched-pairs t-test between best and worst template), consistency (proportion of identical model answers between template pairs), transferability (Intersection-over-Union of the sets of top templates between model pairs), and stability (coefficient of mean deviation of scores across all templates). The t-test and the coefficient of mean deviation do the main work of showing that format effects are real and that newer models are less sensitive to them.","core_discovery":"The paper's central claim is that prompt markup alone is a significant, statistically detectable variable in GPT performance. Using a matched-pairs t-test on the best and worst templates for every model–benchmark pair, the authors find p-values below 0.05 in all but one case (GPT-4-1106-preview on HumanEval), demonstrating that template choice matters. They further claim there is no universally optimal format: GPT-3.5 prefers JSON, GPT-4 favors Markdown, and IoU overlap of top templates between model series is often below 0.2. Larger models are more consistent (above 0.5) and more stable, with GPT-4-1106-preview showing the lowest coefficient of mean deviation, below 0.036 across benchmarks.","pith_inferences":["Editorial inference: the format effect likely generalizes beyond GPT to other instruction-tuned LLMs; if so, cross-paper comparisons that use different templates are not apples-to-apples even when the underlying model is similar.","Editorial inference: the JSON and YAML templates do not preserve wording exactly—they relabel instructions as named fields—so part of the 'format' effect may be an incidental wording effect; a pure-markup control would separate the two.","Editorial inference: a practical evaluation standard would be to run each model on a small portfolio of templates and report the mean and spread, making template sensitivity an explicit, cheap-to-report metric.","Editorial inference: the result suggests a testable prediction about training—models trained on heavily structured or JSON-heavy data might show smaller format effects on those formats, which could be checked with open-weight models."],"forward_implications":["If template choice shifts scores by up to 40%, single-template benchmark results for GPT models are not stable estimates of capability; reporting a range over templates would be more informative.","Prompt format is not transferable across model families: a template tuned for GPT-3.5 may be suboptimal for GPT-4, so prompt engineering must be model-specific.","GPT-4-turbo's greater stability means format sensitivity is partly a model-capability issue, not a fixed property of all LLMs.","Future LLM evaluations should either report performance across a set of formats or explicitly justify the choice of one format, since that choice can change conclusions."],"supporting_citations":[{"why":"supplies the MMLU benchmark used for natural-language reasoning and the multiple-choice example in Figure 1.","marker":"(Hendrycks et al., 2020)"},{"why":"supplies HumanEval, the code-generation benchmark where GPT-4-32k swings over 300% between JSON and plain text.","marker":"(Chen et al., 2021)"},{"why":"supplies CODEXGLUE, the Java-to-C# and C#-to-Java translation tasks where GPT-3.5 shows up to 40% differences.","marker":"(Lu et al., 2021)"},{"why":"supplies HumanEval-X for Java-to-Python translation, another benchmark showing template sensitivity.","marker":"(Zheng et al., 2023)"},{"why":"supplies the FIND benchmark and its string-indicator metric, where switching formats changes scores by 200% for GPT-3.5.","marker":"(Schwettmann et al., 2023)"},{"why":"provides the consistency metric used to measure how often answers match across template pairs.","marker":"(Shu et al., 2023)"},{"why":"prior evidence that minor prompt modifications shift LLM performance; this study extends the question to whole-template structure.","marker":"(Sclar et al., 2023)"},{"why":"prior argument that fixed templates make in-context-learning evaluation inconsistent, which this paper's findings corroborate.","marker":"(Voronov et al., 2024)"}],"fun_headline_variants":["Prompt format alone swings GPT-3.5 scores by 40%","No best prompt template: GPT-3.5 vs GPT-4 differ","Markdown, JSON, YAML: prompt formatting matters","GPT-3.5 varies 40% with prompt template only","Prompt template choice: up to 40% performance gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the four templates are semantically identical, so any performance difference is caused by structure alone; but JSON and YAML rephrase the instructions as labeled fields, which could shift emphasis in a way that is more than purely syntactic.","fun_headline_variants_meta":{"raw":{"variants":["Prompt format alone swings GPT-3.5 scores by 40%","No best prompt template: GPT-3.5 vs GPT-4 differ","Markdown, JSON, YAML: prompt formatting matters","GPT-3.5 varies 40% with prompt template only","Prompt template choice: up to 40% performance gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000148,"raw_usage":{"total_tokens":1151,"prompt_tokens":867,"completion_tokens":284,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":194}},"tokens_in":483,"tokens_out":284,"duration_ms":3330,"temperature":1.0,"reasoning_tokens":194,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:34:53.297151+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a pair of templates that are byte-for-byte identical except for pure markup characters—say, Markdown headers and dashes versus the same words with no markup—and measure the same tasks. If the performance gap collapses to near zero, the reported effect is caused by incidental wording changes, not by structural formatting; if the gap persists, the structural claim holds.","supporting_citations":[],"review_version":1}