{"id":"8fd2b27d-cf8f-4230-99c5-12378acf4b9d","arxiv_id":"2508.13805","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Appending visible descending countdown markers to the prompt makes off-the-shelf LLMs hit exact word or character targets far more often, with exact-match rates reaching 30 to 96 percent across four benchmarks, eleven models, and two languages.","lead":"This paper shows that appending countdown markers to a prompt makes off-the-shelf LLMs hit exact word or character targets far more reliably, with no fine-tuning or extra inference passes. It matters because exact length control usually requires retraining or iterative decoding, and this prompt-only fix works on closed API models at zero extra cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's flagship claim (>95% EM on MT-Bench-LI, quality preserved) is contradicted by Table 4 (74.9% EM; single score 4.839 to 4.063); exact length control is real, but the reported effect size and quality cost are unsupported.","rationale":"I read the paper in good faith: the core contribution is a prompt suffix that externalizes counting, and the data do show large improvements in exact-match rates for many models on random generation, XSUM, MT-Bench-LI, and LIFEBENCH. That effect is real and interesting. The load-bearing problem I see is not the mechanism itself but the paper's own reporting: the abstract's flagship numbers are not the numbers in the tables. A reader relying on the abstract would believe CAPEL exceeds 95% exact match on MT-Bench-LI with no quality loss; Table 4 shows 74.9% and a 0.78-point score drop. The conclusion repeats the same inflated language. This is internal inconsistency, not a disagreement with external consensus, and it directly affects the central claim's credibility. The reader's weakest_assumption about strict marker adherence is related and valid, but I would locate the decisive issue one level up: the claimed effect size on the paper's primary benchmark is unsupported by the paper's own evidence. I would keep the CONDITIONAL verdict because the qualitative effect is well supported, but the conditions must include corrected reporting, error bars or deterministic-seed details, and public postprocessing code so the word-counting procedure can be independently verified. The reproducibility checklist compounds the concern by asserting deterministic decoding while Section 4.3 states temperature 1.0 for OpenAI runs and 0.7 for others; without code or data, the discrepancy cannot be resolved.","tokens_in":16838,"tokens_out":8298,"duration_ms":92352,"concrete_test":"Reproduce the GPT-4.1 row of Table 4 on the 175 MT-Bench-LI instances: run the CAPEL suffix and the naive baseline with the FastChat temperature mapping, strip markers with a postprocessor that replaces each marker by a space (and release that postprocessor), then compute EM and the o4-mini single-answer score. If EM lands near 74.9% and the score near 4.06, the abstract's above-95% and quality-preserved claims are false; if EM exceeds 95%, Table 4 is wrong.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that a one-shot countdown-marker prompt yields exact length control without fine-tuning or iterative sampling, with the flagship demonstration on MT-Bench-LI: strict compliance above 95% and preserved judged quality. For that claim to hold, the tables must show it. They do not. Table 4 reports GPT-4.1 EM rising from 9.7% baseline to 74.9% with CAPEL, while the single-answer score drops from 4.839 to 4.063; the text itself says the score drops by almost one full point. Draft to CAPEL, the quality-recovering variant, reaches only 66.9% EM. On LIFEBENCH Equal-To, Table 6 gives GPT-4.1 an average EM of 39.1%. The only above-95% numbers are XSUM reference-length EM (96.4%) and random-generation EM (94.2%, just below 95%). The abstract and conclusion therefore generalize from the least demanding setting, while the MT-Bench-LI result they cite is over 20 points lower. This matters because the exact-match rate is exactly the promised quantity: the paper claims 'strict length compliance,' not merely improved MALD. The mechanism's fragility compounds the issue: Section 4.6 attributes 87% of CAPEL failures to marker-word fusion and 9% to early stops from safety filters, so EM depends on a delicate behavioral coincidence rather than a guaranteed property. The qualitative finding, that countdown markers sharply improve length compliance on many model-task pairs, is supported by the tables, so this is not a rejection of the method; it is a rejection of the headline numbers as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CAPEL, a prompt-only method that appends descending countdown markers (<N> ... <0>) to a length instruction so that an off-the-shelf LLM writes while counting, aiming for exact token/word/character counts in one pass without fine-tuning or iterative sampling. The method is evaluated on an in-house random-text task, XSUM, MT-Bench-LI, and the LIFEBENCH Equal-To track across eleven API and local models, with metrics for exact-match rate, MAE/MALD, and judged quality, plus a counting diagnostic and an error taxonomy. The central claim is that countdown markers sharply improve exact length compliance across many model-task pairs and that on MT-Bench-LI the flagship GPT-4.1 result exceeds 95% exact match while preserving answer quality.","tokens_in":17117,"tokens_out":3936,"duration_ms":37480,"significance":"A genuinely prompt-only mechanism for near-exact length control would be practically valuable, especially for black-box APIs, and the paper contributes a simple, parameter-free template and an honest failure analysis. The method is not fitted to evaluation benchmarks: the CAPEL suffix appears fixed across tasks, and the headline gains are measured on external benchmarks (XSUM, MT-Bench-LI, LIFEBENCH). The evidence does show large compliance improvements for many model-task pairs, and the error taxonomy (marker-word fusion, safety-filter early stops) is a useful diagnostic. However, the paper's own tables contradict several headline claims, particularly on MT-Bench-LI and on summarization quality preservation, so the significance as stated is not yet established.","major_comments":[{"comment":"The abstract and §4.5 claim that GPT-4.1 strict compliance on MT-Bench-LI exceeds 95%, and §4.5 states EM rises from 9.7% to 96.3%, but Table 4 reports CAPEL EM of 74.9% for GPT-4.1 (baseline 9.7%). The 96.3% figure does not appear in any table for this setting; it matches neither Table 4 nor any other MT-Bench-LI result. The flagship quantitative claim is therefore unsupported by the reported data.","section":"Abstract and §4.5, Table 4"},{"comment":"The text claims CAPEL 'preserves ROUGE-L within ±0.02 of the baseline,' but Table 3 shows larger drops for most models, e.g., GPT-4.1-nano 0.162→0.133 (−0.029), Qwen3-4B 0.174→0.122 (−0.052), and Qwen3-8B 0.177→0.129 (−0.048). The quality-cost claim is thus not supported by the table, and the same issue affects the abstract's assertion that judged answer quality is preserved.","section":"§4.4, Table 3"},{"comment":"The statement that Draft→CAPEL 'outperforms every iterative baseline at the same (one-shot) cost' is contradicted by Table 5: the iterative memory baselines AccMem and Mem achieve single-answer scores of 5.78, higher than Draft→CAPEL's 5.33, and CAPEL's 4.06 is the lowest among the listed GPT-4.1 variants. The claim is only valid for EM, not for quality.","section":"§4.5, Table 5"},{"comment":"The text asserts that 'CAPEL sustains EM > 60% and LS > 90 up to 256 tokens for four of the seven models,' but Figure 3 reports EM values at 256 tokens of 44% (GPT-4.1), 30% (Qwen3-32B-AWQ), 52% (Qwen3-8B), and 0% (GPT-4o-mini), with no model above 60% at that budget. The claimed per-length robustness is not what the figure shows.","section":"§4.6, Figure 3"},{"comment":"The conclusion that CAPEL 'raises exact-match rates from under 30% to above 95%' is only true for the XSUM reference-length condition (e.g., 96.4% for GPT-4.1 in Table 3). On LIFEBENCH the maximum CAPEL EM is 42.6% (Table 6), and on MT-Bench-LI the maximum CAPEL EM is 80.6% (Table 4). The conclusion over-generalizes a single favorable condition to all settings.","section":"§6 Conclusion"}],"minor_comments":[{"comment":"The checklist states that the random-text corpus will be attached only after acceptance and that the code is behind a private GitHub link to be made public later; for a prompt-engineering paper, releasing the exact prompt templates and evaluation scripts with the submission would materially aid reproducibility.","section":"Reproducibility Checklist"},{"comment":"The reference 'Zhang, X.; and ... 2025' is incomplete and cannot be resolved; the three Qwen3 technical report references should be distinguished or merged properly.","section":"References"},{"comment":"Figure 2's x-axis is labeled 'Target length (words/characters),' but LIFEBENCH budgets are specified in tokens; the discrepancy should be clarified in the caption or main text.","section":"§4.6 and Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong core idea and the tables show a real effect, but the abstract and conclusion repeatedly assert numbers the paper's own tables do not support (e.g., >95% EM on MT-Bench-LI when Table 4 says 74.9%). These are not presentation slips; they change the promised effect size and the quality/length trade-off. I believe the issues are fixable by rewriting the claims to match the reported data and by adding appropriate caveats, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: CAPEL, the countdown-marker suffix, seems to do something real. On random text generation, GPT-4.1 exact match goes from 1.9% to 94.2%; on XSUM reference-length, from 72.3% to 96.4%. The qualitative effect holds across most model–task pairs. But the paper's headline claims on MT-Bench-LI are not the numbers in its own tables. The abstract says strict length compliance leaps from below 30% to above 95% and that judged answer quality is preserved. Table 4 reports GPT-4.1 exact match going from 9.7% to 74.9%, with the single-answer score dropping from 4.839 to 4.063. That is a 20-point overstatement and a quality drop of almost a full point. Section 4.5's \"96.3%\" appears nowhere in Table 4. This is not a subtle mismatch.\n\nWhat is genuinely new: a concrete, systematic prompt scaffold — visible descending markers, explicit counting rules, error examples, and a code-aware token rule — evaluated across four benchmarks, eleven models, and two languages. The external-scratchpad rationale is consistent with the counting/CoT literature they cite, and the design is simple enough to be useful in practice for API users who cannot fine-tune. The counting diagnostic in Section 5.1 is a nice, honest addition.\n\nSoft spots, in proportion. First, the reporting. The abstract and conclusion generalize from XSUM and random generation to MT-Bench-LI. Also, Section 4.4 claims ROUGE-L is preserved within ±0.02, but Table 11 shows drops as large as 0.05 for Qwen3-4B. Second, there are no error bars, and the reproducibility checklist says \"deterministic decoding\" while the experimental sections report temperature 1.0 and 0.7 for API runs. That tension needs resolving. Third, the mechanism is fragile: 87% of CAPEL failures come from marker-word fusion (off-by-one shortfalls) and 9% from safety-filter early stops; some small models ignore or refuse the scaffold. Exactness depends on a behavioral coincidence, not a guaranteed property. That is a real limitation, not a fatal flaw. Fourth, the code and data are promised after acceptance, not shipped.\n\nWho this is for: people doing prompt-only constrained generation, especially in production settings with proprietary models. It is not a theory paper and the contribution is modest, but it is useful. The central effect is supported; the headline numbers are not. My recommendation: send it to peer review, then require the authors to align the abstract and prose with their own tables, clarify the determinism question, and release the artifact. A conditional accept after revisions, not a rejection.","headline":"A real prompt-engineering effect, but the abstract's MT-Bench-LI headline numbers are contradicted by the paper's own Table 4, and the quality-preservation claim does not survive contact with the tables.","tokens_in":17755,"tokens_out":2687,"would_cite":false,"duration_ms":28469,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A countdown-marker prompt suffix lets an off-the-shelf LLM hit an exact token count in a single generation, with no fine-tuning or iterative sampling.","keywords":["length-controlled generation","countdown prompting","exact length control","prompt engineering","MT-Bench-LI","LIFEBENCH","chain-of-thought","LLM token counting"],"falsifier":"Re-run the MT-Bench-LI evaluation with GPT-4.1 under the exact CAPEL suffix, count post-processed output tokens, and compare the exact-match rate with the abstract's \"above 95%\" claim and Table 4's 74.9%; a replication that lands near 75% rather than above 95% would show the headline figure is not reproducible as stated. A second check: request a 1000-word neutral passage and parse whether every marker from <1000> to <1> appears exactly once with exactly one word between consecutive markers; the paper's own taxonomy predicts most failures will be fused marker-word pairs, so if the model instead fails by wholesale refusal or random drift, the proposed failure mechanism is wrong.","tokens_in":16531,"feed_emoji":"🎯","tokens_out":7426,"duration_ms":69716,"temperature":0.7,"pith_summary":"The paper's central claim is that exact length control can be achieved through prompt engineering alone: appending a countdown-marker suffix that asks the model to write <N>word, <N-1>word, ... <1>word, <0> turns the output stream into a visible counter, so an off-the-shelf LLM can hit a target token count in one pass. Across open-ended generation, XSUM summarization, MT-Bench-LI instruction following, and LIFEBENCH, the authors report large jumps in exact-match compliance compared with a naive length instruction. The strongest reported gains are on random text generation (from under 2% to 94.2% exact match for GPT-4.1, English) and XSUM (to 96.4% at reference length); on MT-Bench-LI the paper's Table 4 shows GPT-4.1 exact match rising from 9.7% to 74.9%, while the abstract's \"above 95%\" figure does not match that table. If these results hold, strict length control becomes a lightweight prompt technique applicable to closed and open models alike.","feed_headline":"Countdown prompt gives LLMs exact length control","feed_subtitle":"A one-shot marker chain lifts exact-match rates from single digits to 75–96 percent on length-strict benchmarks.","key_machinery":"Countdown-marker scaffold with decrement one: a suffix that demands the output be an explicitly numbered sequence of the form <N>w_N, <N-1>w_{N-1}, ..., <1>w_1, <0>, with the marker value decreasing by exactly one after every emitted word, character, or code line. By externalizing the count into the visible output, generation becomes a serial scratchpad computation, which the paper argues lifts the required computation from constant-depth $\\mathsf{NC}^0$ to $\\mathsf{NC}^1$ and lets iterative counting happen in the open. The single-token decrement is load-bearing: any larger step would require the model to count several unseen tokens before the next marker update and would reintroduce the internal-counting failure regime.","core_discovery":"On the paper's own terms, the discovery is that transformer LLMs fail at length control not because they lack the words but because they cannot maintain an internal counter, and that this internal arithmetic can be replaced by an external, one-step-at-a-time countdown printed in the output. The CAPEL suffix instructs the model to emit exactly the markers <N> down to <1>, each immediately followed by one token and then <0>, with explicit rules that markers are not counted, skips and repeats are forbidden, and nothing follows <0>. Under this prompt the model \"writes while counting,\" reducing each length decision to a local pattern-completion step rather than a global estimate. The paper reports exact-match rates above 90% on several tasks and models, and a diagnostics study in which the same models can count two tokens reliably but fall below 20% accuracy at ten tokens, supporting the claim that the scaffold, not the model's latent counting ability, does the work. On MT-Bench-LI, the paper reports a quality drop for GPT-4.1 (single-answer score 4.839 to 4.063) alongside the compliance gain, which the authors address with a single-pass Draft-to-CAPEL variant that recovers much of the judged quality while keeping one-shot cost.","pith_inferences":["If the mechanism is robust, length control may not be a separate capability but a byproduct of serial bookkeeping; other counting-like constraints, such as exact paragraph counts, exact list-item counts, or \"exactly k examples,\" may yield to the same externalized-countdown pattern.","The paper's own numbers indicate the method is strongest at moderate lengths and decays toward thousands of tokens; a testable extension would be adaptive marker granularity, such as chunk-level markers for long outputs, to push the exact-match boundary past 1K tokens.","The MT-Bench-LI quality drop for GPT-4.1 under CAPEL suggests pure exactness and judged quality can pull in opposite directions; if quality matters more than perfect compliance, the single-pass Draft-to-CAPEL variant may be the practically useful operating point."],"forward_implications":["Exact length control becomes a drop-in prompt: the same CAPEL suffix transfers across proprietary and open-weight models, so systems with hard word or character limits can be built without retraining or multi-pass decoding.","On XSUM and random-text generation, CAPEL pushes exact-match rates above 90% for several models, meaning common summarization and constrained-writing tasks could rely on a single pass rather than revise-until-fit loops.","The scaffold works for both English word counts and Chinese character counts, which suggests the mechanism is about visible enumeration rather than a language-specific tokenizer.","Because the marker sequence is visible, length failures become locally diagnosable: a fused marker, skipped number, or missing <0> tells you exactly where the count went wrong."],"supporting_citations":[{"why":"Supplies the Hansel protocol and XSUM target lengths used to benchmark CAPEL against a training-based length-control framework.","marker":"(Song, Lee, and Ko 2025)"},{"why":"Provides the theoretical result that bounded-depth Transformers cannot count unboundedly, motivating the externalized-countdown design.","marker":"(Feng et al. 2023)"},{"why":"Supports the argument that chain-of-thought-style serial computation lifts Transformers to serial tasks, which the marker output is meant to elicit.","marker":"(Li et al. 2024c)"},{"why":"Defines MT-Bench-LI, the instruction-following benchmark whose length targets CAPEL is evaluated against.","marker":"(Yuan et al. 2024)"},{"why":"Defines LIFEBENCH and its Equal-To track, the longest-range exact-length test used in the paper.","marker":"(Zhang et al. 2025)"},{"why":"Provides the LLM-as-judge single-answer scoring protocol used to measure answer quality on MT-Bench-LI.","marker":"(Zheng et al. 2023)"},{"why":"The BB-MH black-box decoding baseline that CAPEL is compared against on MT-Bench-LI.","marker":"(Gu et al. 2024)"},{"why":"Documents how tokenization affects LLM counting, supporting the diagnostic claim that models cannot count beyond two tokens reliably.","marker":"(Zhang, Cao, and You 2024)"}],"fun_headline_variants":["Countdown prompt nails exact token lengths in one shot","One-shot countdown: LLMs hit 95% exact length compliance","Prompt trick: countdown markers force exact token counts","No fine-tuning needed: countdown prompt controls length","Exact length control via countdown: a one-shot prompt fix"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the model will keep printing the countdown markers in strict descending order, exactly one marker per token, all the way to <0>; if it fuses a marker with a word, repeats or skips a marker, or stops early, the exact-length promise breaks, and the paper's error analysis shows those are precisely the dominant failure modes.","fun_headline_variants_meta":{"raw":{"variants":["Countdown prompt nails exact token lengths in one shot","One-shot countdown: LLMs hit 95% exact length compliance","Prompt trick: countdown markers force exact token counts","No fine-tuning needed: countdown prompt controls length","Exact length control via countdown: a one-shot prompt fix"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000775,"raw_usage":{"total_tokens":3459,"prompt_tokens":1003,"completion_tokens":2456,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":2373}},"tokens_in":619,"tokens_out":2456,"duration_ms":20012,"temperature":1.0,"reasoning_tokens":2373,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:12:14.307202+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the MT-Bench-LI evaluation with GPT-4.1 under the exact CAPEL suffix, count post-processed output tokens, and compare the exact-match rate with the abstract's \"above 95%\" claim and Table 4's 74.9%; a replication that lands near 75% rather than above 95% would show the headline figure is not reproducible as stated. A second check: request a 1000-word neutral passage and parse whether every marker from <1000> to <1> appears exactly once with exactly one word between consecutive markers; the paper's own taxonomy predicts most failures will be fused marker-word pairs, so if the model instead fails by wholesale refusal or random drift, the proposed failure mechanism is wrong.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Hansel protocol and XSUM target lengths used to benchmark CAPEL against a training-based length-control framework."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the theoretical result that bounded-depth Transformers cannot count unboundedly, motivating the externalized-countdown design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines MT-Bench-LI, the instruction-following benchmark whose length targets CAPEL is evaluated against."},{"cited_title":"P.; Gonzalez, J","cited_arxiv_id":null,"evidence_quote":"Provides the LLM-as-judge single-answer scoring protocol used to measure answer quality on MT-Bench-LI."}],"review_version":2}