{"id":"df13c3c7-c0f1-4dd8-8943-85cb3e2e711d","arxiv_id":"2506.10716","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper reports that a concise optimized prompt reduces reasoning tokens by up to 87.5% while roughly preserving accuracy on GSM8K, SVAMP, and MATH-500.","lead":"This paper introduces PREMISE, a prompt-only method that aims to make large reasoning models solve math problems with far fewer output tokens by instructing them to write concise step-by-step solutions. The intended payoff is cheaper and faster commercial AI inference, but the paper's own tables contradict several headline accuracy and cost numbers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4.2 contradicts Table 1 on Gemini MATH-500 accuracy (82% vs 96%), and the abstract reports yet another number (92%); the headline match-or-exceed claim has no internally consistent supporting result.","rationale":"The single most load-bearing concern is the internal inconsistency of the reported measurements. The abstract's core quantitative promise is that PREMISE matches or exceeds baseline accuracy while cutting token cost by 69–82%. That promise is refuted, or at least unverifiable, when Table 1 and Section 4.2 give irreconcilable accuracy values for the same model, dataset, and method: 96% versus 82%. If the 82% figure is correct, the claim 'match or exceed baseline accuracy' is false for Gemini on MATH-500, one of the three benchmark datasets; if the 96% figure is correct, the paper's own limitation section is inexplicable. The abstract's third number (92%) compounds the problem. This is not a modeling assumption but a factual contradiction in the primary evidence, so the central claim cannot be accepted as written. The reader's verdict of REJECT is therefore upheld, though the reader's weakest_assumption focused on the premise that a static prompt can redirect hidden reasoning traces; my concern sits one level earlier, at the consistency of the experimental record itself. A clean re-run of the Gemini/MATH-500 cell would settle the matter, and disclosure of the optimization protocol would make the result interpretable. Until then, the paper's main contribution is not established.","tokens_in":11062,"tokens_out":4215,"duration_ms":46727,"concrete_test":"Run the PREMISE prompt from Figure A.1 on MATH-500 with Gemini-2.5-flash-preview-04-17 using the same sampling settings as in Table 1; record accuracy, thinking tokens, completion tokens, input tokens, and cost. Compare the output to both Table 1 and Section 4.2. If accuracy is approximately 82%, Table 1's 96% is wrong and the match-or-exceed claim fails on that setting; if accuracy is approximately 96%, Section 4.2's stated 14% drop is contradicted. Either way, also publish the exact prompt, the optimization trajectory, and the number of optimization iterations so the result is reproducible.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that PREMISE matches or exceeds baseline accuracy while cutting reasoning tokens and dollar cost. The paper's own data for Gemini-2.5-flash on MATH-500 are mutually inconsistent: Table 1 lists PREMISE accuracy as 96% with Normal at 95%, but Section 4.2 states 'PREMISE attains only 82% accuracy on MATH-500 with Gemini, a 14% drop relative to the normal CoT run.' The abstract reports a different number again ('91%→92%'). These values cannot be reconciled by sampling noise or a small typo: 82% is a 14-point deficit, not a 1-point drop, and 96% is a 1-point gain. Because the headline efficiency claim depends on the accuracy column being correct, this direct contradiction means the primary experimental result is not established as reported. The same table also shows OpenAI o1 cost per iteration increasing roughly 3x on GSM8K ($0.0228→$0.0706) and 23% on MATH-500, which the abstract's 69–82% cost-cut range does not mention; the claimed savings are model-dependent. Section 4.2 even acknowledges this exception, yet the abstract and conclusion present the result as universal. Without a corrected, reproducible Table 1 and Section 4.2, the central claim is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PREMISE, a prompt-only framework intended to reduce chain-of-thought reasoning cost in black-box large reasoning models. It defines trace-level overthinking and underthinking metrics, uses them together with a multi-objective natural-language gradient search to optimize a static prompt, and evaluates the resulting prompt on GSM8K, SVAMP, and MATH-500 with Claude 3.7 Sonnet, OpenAI o1, and Gemini 2.5 Flash, as well as in a multi-agent setting. The advertised result is that PREMISE matches or exceeds baseline accuracy while reducing reasoning tokens by up to 87.5% and dollar cost by 69–82%. The manuscript is internally inconsistent on the central numbers: the abstract, Table 1, and Section 4.2 report mutually incompatible Gemini MATH-500 accuracy values, and the claimed universal cost reduction is contradicted by the same table for OpenAI o1. The method section also relies on quantities that are not computable as defined and does not state whether the optimized prompt was selected on a held-out split.","tokens_in":11394,"tokens_out":4952,"duration_ms":57389,"significance":"If the empirical claims were correct, PREMISE would be a practically valuable result: a single-pass, black-box, prompt-only method that cuts LRM inference cost by roughly four-fifths on standard math benchmarks without accuracy loss would be directly usable with commercial APIs. The idea of combining trace-level diagnostics with textual multi-objective search is a reasonable direction, and the paper includes experiments across several models and an agent setting. However, the central claim is not currently supported by the manuscript's own evidence. The numerical contradictions in the headline results and the lack of a clearly described held-out evaluation protocol mean that the reported savings cannot be trusted as stated. I also see no code or detailed prompt-release that would allow independent verification, which further lowers the evidentiary value of the reported numbers.","major_comments":[{"comment":"The central experimental claim is internally inconsistent. The abstract reports '91%→92% with Gemini' on MATH-500, Table 1 lists the PREMISE accuracy for Gemini-2.5-flash on MATH-500 as 96% (with Normal at 95%), and Section 4.2 states that 'PREMISE attains only 82% accuracy on MATH-500 with Gemini, a 14% drop relative to the normal CoT run.' These three values cannot be reconciled by sampling noise or a typo: 82% is a 14-point deficit, 96% is a 1-point gain, and 92% is yet another value. Because the 'match or exceed baseline accuracy' claim is the load-bearing part of the paper, this contradiction means the primary result is not established as reported.","section":"Abstract, Table 1, Section 4.2"},{"comment":"The claimed cost reduction is contradicted by the paper's own data for OpenAI o1. Table 1 shows that on GSM8K, PREMISE increases thinking tokens from 249 to 1,012 and per-iteration cost from $0.02280 to $0.07061, and on MATH-500 it increases cost from $0.10938 to $0.13457. Section 4.2 explicitly acknowledges this exception, yet the abstract and conclusion state that PREMISE cuts dollar cost by 69–82% without qualification. The claimed savings are model-dependent, and the abstract's universal formulation is not supported by the reported experiments.","section":"Table 1, Section 4.2, Abstract"},{"comment":"The overthinking and underthinking metrics are not operational as defined. L*(q) in Section 3.3 is defined as the minimum length over the set R of all possible reasoning traces, and k*(r,q) in Section 3.4 requires knowing whether a correct continuation exists for an arbitrary prefix in R. For any realistic language model, R is effectively infinite and the existence of a correct continuation is undecidable or at least requires an oracle. The paper gives no approximation, estimator, or computational procedure for these quantities, so the claimed trace-level diagnostics cannot actually be computed from API outputs. This undermines the methodological foundation of the framework, not just its presentation.","section":"Sections 3.3 and 3.4"},{"comment":"The evaluation protocol does not establish that the reported results are out-of-sample. The multi-objective search in Section 3.6 directly optimizes the two quantities that are later reported as gains: accuracy and token length. Nowhere in Section 4.1 or Section 4.2 is it stated whether the optimized prompt was selected on a held-out portion of each benchmark, whether the numbers in Table 1 come from the same examples used during optimization, how many optimization runs were performed, or what the variance across runs was. Without a held-out evaluation, the reported token reductions and accuracy matches may simply reflect fitting to the test set. This is a load-bearing omission for any prompt-optimization paper.","section":"Section 3.6 and Section 4.1"},{"comment":"Additional numerical inconsistencies appear in the multi-agent results. Section 5.3 states that 'Gemini's cost on MATH-500 falls by nearly 70%,' but Table 2 shows Gemini on MATH-500 going from $0.146 to $0.085, which is a 42% reduction, not nearly 70%. Section 5.3 also says that on GSM8K with Claude, PREMISE trims 'more than 1.1k reasoning tokens per problem' and the footnote gives a 19% reduction, which is not consistent with the text's cost-focused framing. These errors compound the impression that the empirical tables and prose were not checked against each other.","section":"Section 5.3 and Table 2"}],"minor_comments":[{"comment":"There is a typo in 'gradient-base prompt optimization method'; it should be 'gradient-based'.","section":"Section 3.6"},{"comment":"The dataset name is inconsistently written as 'SV AMP' in the abstract and body and 'SVAMP' in the table captions and references; please standardize.","section":"Throughout"},{"comment":"The appendix references 'Figure A.1: PREMISE Generated Efficient Reasoning Prompt,' but the figure content is missing from the manuscript. The actual prompt is central to the method and should be printed in full.","section":"Appendix A"},{"comment":"The notation q(x_i) in the accuracy definition is confusing: q is described as a reasoning schema, but earlier q denotes a question. Please use distinct symbols for questions and prompts.","section":"Section 3.1"},{"comment":"The text says 'Under the overthinking metric defined in Section 3.4,' but the overthinking metric is defined in Section 3.3; the underthinking metric is in Section 3.4.","section":"Section 5.1"},{"comment":"The statement that for OpenAI 'the accuracy of PREMISE is still slightly higher' is contradicted by Table 1, where PREMISE is below Normal on MATH-500 (97% vs. 98%) and below Normal on SVAMP (96% vs. 97%).","section":"Section 5.2"}],"recommendation":"reject","confidential_remarks":"The central problem is not a difference of opinion with the field's consensus; it is that the paper's own reported numbers contradict each other on the exact quantities that constitute the headline contribution. The abstract, Table 1, and Section 4.2 cannot all be correct, and the o1 cost increases directly contradict the abstract's unqualified 69–82% cost-cut claim. In addition, the method section defines metrics that are not computable and the evaluation section does not state whether prompt selection was out-of-sample. These issues cannot be fixed with local edits; they require redoing the experiments with a clearly specified evaluation protocol. I would not invite resubmission of the current version, though a corrected and properly validated version of the idea could be submitted as new work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me save you the trouble: the headline result as stated is not supported by the paper's own data. Table 1 gives PREMISE 96% on Gemini MATH-500; Section 4.2 says it drops to 82%; the abstract says 91 to 92. Those are not reconcilable. The same table shows o1 cost rising roughly 3x on GSM8K and 23% on MATH-500, which the abstract's flat 69-82% cost-cut range does not mention. So the match-or-exceed accuracy claim fails as reported.\n\nWhat is genuinely useful: this is a systematic attempt to shrink reasoning tokens through prompt-only control, with token and dollar-cost numbers across three models and three datasets, plus an extension to a multi-agent system. The authors are candid about the o1 exception and the Gemini MATH-500 degradation in Section 4.2; they do not hide those failures. The multi-agent results, while noisy, suggest concise prompts can reduce cost while roughly preserving accuracy in some settings. That part is worth a look once the numbers are cleaned up.\n\nThe soft spots are not minor. The trace-level over/underthinking diagnostics in Section 3 rely on L*(q) and k*(r,q), which are not available in a black-box setting and are never computed. The optimization section describes a scalarized textual gradient, but not prompt initialization, iteration count, or whether the final prompt was selected on the test set. That makes the reported reductions look fitted rather than predictive. The citation pattern is honest: Chain-of-Draft, Sketch-of-Thought, Token-Budget, and CCoT are cited, and they already show that length instructions cut tokens. The claimed novelty of trace diagnostics guiding prompt optimization is not demonstrated.\n\nIf I were editing, I would not desk-reject this outright, because the problem is relevant and the multi-agent data could be useful, but I would send it back for major revision and insist on a consistent table, a held-out prompt-selection protocol, and evidence that the diagnostics actually drive the optimization. As it stands, no one should cite the 87.5% token reduction or 69-82% cost figures.","headline":"The paper's headline savings claim is contradicted by its own Table 1, Section 4.2, and abstract, so the central result is not established; the concise-CoT phenomenon itself is real but already covered by prior work.","tokens_in":11883,"tokens_out":3229,"would_cite":false,"duration_ms":38821,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A prompt-only method matches math accuracy while cutting reasoning tokens by up to 87.5%.","keywords":["prompt optimization","chain-of-thought","large reasoning models","overthinking","underthinking","token efficiency","mathematical reasoning","black-box API"],"falsifier":"Re-run PREMISE on MATH-500 with Gemini-2.5-flash and compare the measured accuracy against the paper's own numbers: the prose reports 82%, while Table 1 reports 96%.","tokens_in":1695,"feed_emoji":"🧮","tokens_out":2903,"duration_ms":84990,"temperature":0.7,"pith_summary":"The paper tries to prove that the long chain-of-thought traces produced by large reasoning models are not a necessary price of accuracy. It introduces PREMISE, a prompt-only framework that measures where traces waste tokens and then optimizes a prompt so the model reasons more tersely while staying correct. Across three math benchmarks and two commercial models, the method matches or improves accuracy while shrinking reasoning tokens by as much as 87.5% and cutting dollar cost by 69–82%. If correct, this makes efficient reasoning available to anyone using a black-box API, with no fine-tuning and no multi-sample decoding.","feed_headline":"One prompt cuts math-reasoning tokens by 87.5%","feed_subtitle":"Prompt-only PREMISE matches Claude/Gemini accuracy on GSM8K, SVAMP, and MATH-500 while cutting API cost 69–82%.","key_machinery":"The central machinery is the pair of trace-level inefficiency metrics $\\Xi_O$ and $\\Xi_U$, measuring overthinking and underthinking respectively, combined with a multi-objective natural-language gradient search. Textual gradients for answer correctness and token length are scalarized as $\\delta = \\lambda \\delta_{acc} + (1-\\lambda)\\delta_{len}$ and applied as prompt updates, so the method navigates the Pareto frontier between accuracy and brevity entirely through a single-pass black-box interface.","core_discovery":"The paper's central claim is that a static, optimized prompt can compress the hidden chain-of-thought of commercial black-box reasoning models without touching model weights. PREMISE defines overthinking as the fraction of tokens in a correct trace that exceed the shortest correct trace, and underthinking as the fraction of a trace after the point where it irreversibly leaves a recoverable correct path; it uses these diagnostics in a multi-objective textual search that balances correctness against token length. On GSM8K, SVAMP, and MATH-500, the resulting prompt matches or exceeds standard CoT accuracy for Claude and Gemini while reducing reasoning tokens by up to 87.5% and dollar cost by 69–82%. The paper reports that this is not universal: with OpenAI o1 the thinking tokens and cost rise, and its own prose reports an accuracy drop to 82% for Gemini on MATH-500, a number the table contradicts.","pith_inferences":["A natural extension is difficulty-conditioned prompts: estimate each problem's complexity and choose the compression strength $\\lambda$ accordingly, which could recover Gemini's MATH-500 accuracy while retaining most token savings.","Since the overthinking and underthinking metrics require access to tokenized reasoning traces, the same diagnostic idea could be adapted to fully closed APIs by using final-answer consistency or completion-length distributions as proxy signals.","The reported discrepancy for Gemini on MATH-500 (82% in the prose versus 96% in the table) is worth resolving before relying on the method's accuracy guarantees; a simple rerun of that single benchmark would settle which number represents the method.","Because the framework is prompt-only, it could be composed with weight-accessible efficiency techniques in settings where fine-tuning is allowed, potentially multiplying the savings."],"forward_implications":["On GSM8K and SVAMP with Claude and Gemini, PREMISE cuts total token footprint by roughly 75–79% with accuracy within one point, and dollar cost by 69–82%.","The same prompt-level compression works inside multi-agent systems: for example, Gemini on GSM8K rises from 85% to 90% accuracy while cost falls by 37%.","On models that expose a hidden reasoning channel, the prompt can redirect and compress the trace; on OpenAI o1, which exposes only a single completion stream, PREMISE increases thinking tokens and cost, so the paper expects limited leverage until reasoning usage statistics are exposed.","The compression ratio is not universally safe: on proof-heavy MATH-500 with Gemini, an overly concise prompt can skip justifications and drop accuracy by about 14 points, so a difficulty-adaptive token budget is needed.","Because the method needs only one forward pass per question and no reranking or decoding overhead, it is deployable in latency-sensitive and API-constrained settings."],"supporting_citations":[{"why":"Supplies the natural-language gradient optimization procedure that PREMISE uses to update its prompt.","marker":"Zhang et al. 2024"},{"why":"Defines chain-of-thought prompting, the baseline behavior that PREMISE compresses.","marker":"Wei et al. 2022"},{"why":"Provides the GSM8K benchmark used in the main evaluations.","marker":"Cobbe et al. 2021a"},{"why":"Provides the SVAMP benchmark used in the main evaluations.","marker":"Patel et al. 2021"},{"why":"Provides the MATH-500 benchmark, the proof-heavy set where compression can fail.","marker":"Lightman et al. 2024"},{"why":"Represents the token-budget prompting baseline that PREMISE is compared against.","marker":"Han et al. 2024"},{"why":"Represents the Chain-of-Draft baseline for concise reasoning.","marker":"Xu et al. 2025b"},{"why":"Represents the concise chain-of-thought prompting baseline.","marker":"Renze and Guven 2024"},{"why":"Represents the Sketch-of-Thought baseline used in the comparisons.","marker":"Aytes et al. 2025"},{"why":"Supplies the Promptor multi-agent system in which PREMISE is also evaluated.","marker":"Chen et al. 2025"}],"fun_headline_variants":["Prompt-only method cuts CoT tokens up to 87.5%","Cuts API math cost 69-82% without accuracy loss","PREMISE: same accuracy, 87.5% fewer reasoning tokens","Reasoning tokens cut 87.5% on Claude/Gemini","Prompt-level fix: 87.5% fewer tokens, 69-82% cheaper"],"cache_read_input_tokens":13952,"weakest_assumption_plain":"The method assumes that a single static prompt, optimized on held-out traces, can make any black-box reasoning model produce hidden reasoning that is shorter yet still correct.","fun_headline_variants_meta":{"raw":{"variants":["Prompt-only method cuts CoT tokens up to 87.5%","Cuts API math cost 69-82% without accuracy loss","PREMISE: same accuracy, 87.5% fewer reasoning tokens","Reasoning tokens cut 87.5% on Claude/Gemini","Prompt-level fix: 87.5% fewer tokens, 69-82% cheaper"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000979,"raw_usage":{"total_tokens":4170,"prompt_tokens":970,"completion_tokens":3200,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":3100}},"tokens_in":586,"tokens_out":3200,"duration_ms":28896,"temperature":1.0,"reasoning_tokens":3100,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:19:52.685261+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run PREMISE on MATH-500 with Gemini-2.5-flash and compare the measured accuracy against the paper's own numbers: the prose reports 82%, while Table 1 reports 96%.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MATH-500 benchmark, the proof-heavy set where compression can fail."}],"review_version":1}