{"id":"c52e7ee7-a05d-4e47-a32e-b7e4b2610377","arxiv_id":"2502.04295","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Jointly optimizing prompt wording and formatting with UCT-guided format search and LLM-generated formats improves accuracy over content-only prompt optimizers on several benchmarks and models.","lead":"This paper introduces CFPO, a prompt-optimization method that searches over both the wording and the layout of prompts, and reports accuracy gains on math, reasoning, and classification benchmarks across four open-source language models. It matters because prompt format is often treated as fixed, and the results suggest jointly tuning content and format can improve LLM performance beyond content-only methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The joint-format advantage is confounded by evaluation budget: CFPO evaluates many more prompt candidates per content candidate than content-only baselines, so the reported gains may reflect more search rather than format optimization.","rationale":"The reader's identified weakness, small evaluation subsets and missing variance reporting, is valid and directly related. My stress-test reading adds a more structural confound: CFPO spends additional evaluation budget on format search, so the advantage over content-only variants is not cleanly attributable to format as an optimization dimension. This is a correctness risk rather than a claim of unfairness; it is a missing control. The paper's own stability analysis only repeats CFPO on GSM8K and does not compare against baselines under equal compute. A budget-matched rerun is feasible because the code is available and the evaluation metric is exact match. If CFPO still outperforms CFPOc and ProTeGi under matched target-model calls, the central claim would be substantially stronger. Until then, the conditional verdict is appropriate, with the condition made explicit: demonstrate that the improvement survives equal evaluation budget and larger selection subsets.","tokens_in":21553,"tokens_out":5477,"duration_ms":63084,"concrete_test":"Run a budget-matched comparison: instrument total target-model forward passes or API-equivalent cost for CFPO, CFPOc, CFPOc+f, ProTeGi, and SAMMO on GSM8K and Big-Bench. Then re-run CFPOc and ProTeGi with enough additional iterations or content candidates to consume the same total target-model evaluations as full CFPO per seed, selecting the final prompt on the same validation subset. If CFPOc or ProTeGi then matches CFPO's test accuracy within sampling error, the claimed joint-format benefit is not established. As a second check, repeat the Big-Bench comparison at selection-subset sizes of N=50 and N=500 over at least 5 seeds, reporting selection-set and test-set scores separately; if the CFPO advantage shrinks at N=500 or falls within the bootstrap confidence interval of the content-only baseline, the headline result is driven by selection noise rather than format optimization.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that jointly optimizing content and format beats content-only optimization rests on Table 1 and the ablation in Table 2. The most insecure premise is that CFPO and its comparators are compared under a comparable search budget. In Algorithm 1, CFPO evaluates each content candidate under 2k formats (k UCT-selected plus k LLM-generated; k=4 in Appendix B.1) on the selection dataset, for every content candidate in every round. The CFPOc ablation, by definition, holds the format fixed and therefore consumes none of those format evaluations. Table 7 reports CFPO's per-round cost (237 API calls, roughly $2.69) but no comparable budget or cost is reported for CFPOc, ProTeGi, or SAMMO. If the content-only variants are simply given fewer target-model evaluations, the gaps in Table 2 and Table 1 can be explained by additional search effort on a small selection set rather than by the format dimension itself. On the 50-example Big-Bench task this is especially acute: choosing the best among many extra candidates on a 50-item binomial metric amplifies selection noise, and the reported 8-10 point gaps are within roughly two standard errors. The reproduced CFPO-only standard deviations in Table 6 do not address baseline variance or budget matching. The paper has real strengths, including a clear method, multiple target models, and an available codebase, but the comparison that would isolate the format contribution is missing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CFPO, an iterative prompt optimization method that jointly optimizes prompt content and formatting. It decomposes prompts into content and format components, mutates content via case diagnosis and Monte-Carlo sampling, and explores formats through a dynamic pool scored by UCT selection plus LLM-generated formats. Experiments on Big-Bench Classification, ARC-Challenge, GSM8K, and MATH500 across four open-source LLMs report consistent improvements over content-only baselines such as ProTeGi and SAMMO, with ablations attributing gains to format generation, UCT selection, and joint content-format optimization.","tokens_in":21812,"tokens_out":3186,"duration_ms":31656,"significance":"If the reported results are reliable, the paper makes a useful contribution by treating prompt format as a searchable, model-dependent dimension rather than a fixed design choice. The methodology is clearly described, the code is released, and the evaluation includes multiple target models and tasks. The paper also provides useful stability and cost analyses (Tables 6 and 7) and a format-generation-model ablation (Table 9). However, the significance of the central claim is currently limited by two methodological gaps: the comparison between CFPO and content-only baselines is not matched on evaluation budget, and the small selection subsets (especially 50 examples for Big-Bench) make the reported differences statistically fragile. These issues can be addressed with additional experiments and uncertainty quantification.","major_comments":[{"comment":"The comparison between CFPO and content-only baselines is confounded by evaluation budget. In Algorithm 1, CFPO evaluates each content candidate under 2k formats (k=4, so 8 evaluations) per round, while content-only baselines such as ProTeGi and SAMMO evaluate a single format. Table 7 reports CFPO's per-round cost of 237 API calls, but no comparable budget or cost is reported for CFPOc, ProTeGi, or SAMMO. Consequently, the improvements in Tables 1 and 2 may reflect additional target-model evaluations rather than the format dimension itself. Please provide a matched-budget comparison, for example by allowing content-only baselines the same number of evaluations per content candidate or by reporting CFPOc with an equivalent evaluation budget.","section":"Section 3.3.3 (Algorithm 1) and Table 7"},{"comment":"The final prompt is selected as the argmax on a very small evaluation subset: 50 examples for Big-Bench Classification, 500 for GSM8K, 500 for ARC-C, and 300 for MATH500. On the 50-example task, the reported gaps (e.g., 94 vs. 86 for Mistral-7B in Table 1) are within roughly two standard errors of a binomial proportion, so the ranking of methods may be noise. No confidence intervals, significance tests, or repeated baseline runs are reported; Table 6 reports standard deviations for CFPO only. To support the claim of consistent improvements, please report variance or confidence intervals for all methods, or use larger selection sets.","section":"Section 4.1 and Section B.3"},{"comment":"The ablation comparing full CFPO with the sequential variant CFPOc+f does not isolate the benefit of joint optimization. CFPOc+f performs 'a separate, single-step format optimization,' whereas full CFPO performs format optimization in every iteration with 2k format evaluations per content candidate. Without matching the number of format evaluations or optimization iterations, the gap between full CFPO and CFPOc+f may simply reflect a larger total search budget. To establish that joint optimization is superior to sequential optimization, the sequential variant should be given the same evaluation budget as the full pipeline.","section":"Table 2 and Section 4.3"}],"minor_comments":[{"comment":"The subsection title 'Prefix Tunning' contains a typo; it should read 'Prefix Tuning.'","section":"Section 2.2"},{"comment":"In the Monte-Carlo Sampling meta-prompt for examples, 'shoud' should be 'should'.","section":"Appendix A.2.2"},{"comment":"The references to Voronov et al. 2024a and 2024b refer to the same paper published in two venues; please consolidate them into a single citation to avoid duplication.","section":"Related Work"},{"comment":"The caption of Table 5 repeats the caption of Table 3 ('Ablation of format generation and comparison of format selection strategies') but the table reports MMLU results; the caption should describe the MMLU experiment.","section":"Table 5"},{"comment":"The text says optimization uses subsets sampled from the 'training split' while Section B.3 refers to an 'evaluation set' and then reports results on a 'test set.' Please clarify the terminology and whether the reported numbers are on held-out test sets.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a sensible idea and a real empirical gap. The joint content-format optimizer is a reasonable engineering contribution, and the ablations show the full method beats its own parts. But the main comparison against content-only baselines is confounded by evaluation budget, and the small selection sets make the headline gaps fragile. With budget-matched runs and confidence intervals, the central claim might hold—but the paper as written does not demonstrate it.\n\nWhat's new: integrating UCT-based format search with LLM-generated formats into an iterative content optimizer. That is genuinely new relative to prior cited work, and the structured template is a practical way to decompose prompts. The ablations (format-only, content-only, sequential) are the right experiments to run, and they show the full pipeline helps on two tasks. The stability runs in Table 6 show CFPO itself is fairly reproducible, and the code is public.\n\nThe soft spot is the one the stress-test note flags. Full CFPO evaluates each content candidate under 8 formats (4 UCT-selected, 4 generated) every round; the content-only ablation holds format fixed and does one evaluation. So CFPO gets roughly 8x the target-model evaluations per content candidate. The paper reports cost for CFPO (Table 7) but never reports the evaluation budget for ProTeGi, SAMMO, or CFPOc. Without that, the gaps in Tables 1 and 2 could reflect extra search effort rather than the format dimension. That is a load-bearing comparison, so it needs to be fixed.\n\nThe second issue is statistical. The selection set for Big-Bench is 50 examples. Matching that with the reported 8-point gap, the standard error is about 6 points, so the gap is not clearly above noise. No confidence intervals or repeated baseline runs appear in Table 1. The authors do report CFPO variance in Table 6, but not for the baselines, so we can't tell if the ranking is stable.\n\nThese are fixable empirical problems, not conceptual ones. The method is coherent and the paper is honest about limitations. I'd send it to a serious referee, with the request that the authors add budget-matched baselines and either CIs or multiple seeds. Who benefits: practitioners who want a practical format-search tool, and researchers working on prompt optimization. I'd bring it to a reading group as a case study in evaluation confounds.","headline":"The format-optimization idea is sensible and practical, but the evidence for its superiority over content-only search is currently undercut by an evaluation-budget imbalance and noisy 50-example selection sets.","tokens_in":22360,"tokens_out":3592,"would_cite":true,"duration_ms":33928,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Jointly optimizing a prompt's content and its formatting—layout, labels, separators—improves LLM accuracy beyond optimizing content alone.","keywords":["prompt optimization","prompt formatting","content-format joint optimization","LLM sensitivity","UCT format search","few-shot prompting","automatic prompt engineering"],"falsifier":"Run CFPO and a content-only baseline on the 50-example Big-Bench task multiple times with different seeds and report the confidence interval of the chosen prompt's test accuracy; if the CFPO advantage is not consistently outside the noise, the joint-optimization claim fails there. A complementary test is to give the content-only baseline the same number of format evaluations per round and check whether the gap shrinks.","tokens_in":21340,"feed_emoji":"🎯","tokens_out":6117,"duration_ms":51121,"temperature":0.7,"pith_summary":"The paper argues that the visual and structural format of a prompt—how instructions, examples, and queries are labeled, ordered, and separated—is a searchable dimension that content-only prompt optimizers leave on the table. It introduces CFPO, an iterative method that mutates prompt content while separately exploring a pool of prompt renderers and query formats, selecting the best combination for each candidate. Across four benchmarks and four open-source language models, CFPO reports higher accuracy than content-only baselines, with the largest gains on reasoning tasks. If correct, prompt-format search is a cheap, model-agnostic lever for improving LLM performance without changing the model.","feed_headline":"Joint content-format search beats content-only prompts","feed_subtitle":"CFPO tunes layout and wording together, lifting GSM8K accuracy by up to 7.5 points over content-only baselines.","key_machinery":"The central object is the structured prompt template, which splits a prompt into content components (Task Instruction, Task Detail, Output Format, Few-shot Examples, Query) and format components (Prompt Renderer and Query Format). CFPO carries out optimization as a dual loop: a content optimizer proposes mutated candidates through case-diagnosis and Monte-Carlo sampling, and a format optimizer assigns each candidate the best format from a dynamically growing pool, using an Upper Confidence Bounds (UCT) score to balance trying new formats against reusing well-scored ones, with new formats generated by an LLM from descriptions and rendered code. This decoupling is what lets the method search the two dimensions separately while evaluating them together.","core_discovery":"CFPO's claim is that treating prompt content and prompt format as interdependent and optimizing them jointly yields measurable gains over optimizing content alone. The optimizer alternates between a content stage—which revises individual components (task instruction, task detail, output format, few-shot examples) using both correct and incorrect cases plus Monte Carlo sampling—and a format stage, which scores and refines a dynamic pool of format configurations under a UCT selection rule. In the paper's experiments, full CFPO outperforms its own content-only, format-only, and sequential variants, and beats ProTeGi and SAMMO on nearly every model-task pair, for example reaching 53.22 on GSM8K with Mistral-7B versus ProTeGi's 45.72.","pith_inferences":["An untested implication is that the optimal format for a task may transfer across models of the same family or even across tasks; the format pool design makes this plausible, but the paper does not report cross-model format transfer.","A matched-budget comparison would be needed to confirm that the gains come from content-format coupling rather than from CFPO spending extra evaluations per round on format candidates.","On the 50-example Big-Bench split, the reported 8-point gap over SAMMO is within the standard error of about 5 points; repeated runs with confidence intervals are the natural way to test whether the joint-optimization advantage is real on small evaluation sets."],"forward_implications":["Prompt formatting should be reported and treated as a variable in future prompt-optimization and evaluation studies, since it can change accuracy by several points on the same prompt content.","Pre-trained (base) models may benefit systematically from longer prompts and more in-context examples, whereas instruction-tuned models need less explicit scaffolding, which could guide default prompt designs.","Because format optimization appears to converge faster than content optimization, a schedule that spends more format-search budget early and content budget later could lower optimization cost.","CFPO's gains do not require a top-tier optimizer model: using Qwen2.5-14B as the optimizer still improves over content-only baselines, suggesting the approach transfers to cheaper setups."],"supporting_citations":[{"why":"ProTeGi is the feedback-driven content-only baseline CFPO must beat; it supplies the beam-search mutation loop the authors build on.","marker":"(Pryzant et al., 2023)"},{"why":"SAMMO is the strongest structured baseline with a predefined format pool; beating it motivates the dynamic format search.","marker":"(Schnabel and Neville, 2024)"},{"why":"Establishes that separators and spacing alone can shift LLM accuracy, providing the empirical premise that format is worth optimizing.","marker":"(Sclar et al., 2024)"},{"why":"Shows no single prompt format wins universally across GPT-based models, motivating per-content format search.","marker":"(He et al., 2024)"},{"why":"Shows few-shot example formatting affects in-context learning performance, guiding the query-format component choices.","marker":"(Voronov et al., 2024b)"},{"why":"Supplies the UCT bandit rule used to balance format exploitation and exploration in the dynamic format pool.","marker":"(Kocsis and Szepesvári, 2006)"},{"why":"Provides the GSM8K benchmark where CFPO reports its largest gains over content-only baselines.","marker":"(Cobbe et al., 2021)"}],"fun_headline_variants":["CFPO tunes prompt layout and wording together for 7.5-point gains","Prompt layout is as critical as text: CFPO beats content-only","Co-tuning prompt content and format outperforms content-only on GSM8K","Joint content-format optimization lifts LLM accuracy on GSM8K"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's success rests on the assumption that a prompt's score on the small evaluation subset (as few as 50 examples) reliably predicts which prompt will generalize best to the test set.","fun_headline_variants_meta":{"raw":{"variants":["CFPO tunes prompt layout and wording together for 7.5-point gains","Prompt layout is as critical as text: CFPO beats content-only","Co-tuning prompt content and format outperforms content-only on GSM8K","Joint content-format optimization lifts LLM accuracy on GSM8K"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001298,"raw_usage":{"total_tokens":5248,"prompt_tokens":851,"completion_tokens":4397,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":4319}},"tokens_in":467,"tokens_out":4397,"duration_ms":30286,"temperature":1.0,"reasoning_tokens":4319,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T22:51:09.373871+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CFPO and a content-only baseline on the 50-example Big-Bench task multiple times with different seeds and report the confidence interval of the chosen prompt's test accuracy; if the CFPO advantage is not consistently outside the noise, the joint-optimization claim fails there. A complementary test is to give the content-only baseline the same number of format evaluations per round and check whether the gap shrinks.","supporting_citations":[],"review_version":1}