{"id":"f3bd3032-c0e6-40c9-ae5d-07aec581e06c","arxiv_id":"2505.16234","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LIFEBench's evaluation of 26 LLMs shows most follow short length instructions but degrade sharply beyond a few hundred words, and none reliably hit vendor-claimed maximum output lengths.","lead":"This paper introduces LIFEBench, a benchmark of 10,800 tasks that tests whether large language models can follow length instructions, from 16 to 8,192 words, in English and Chinese. The results show most models obey short length requests but fall far short on long ones, often below what their makers claim they can produce.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The LS scoring curve's hand-chosen k1=5,k2=2 is the load-bearing hinge: all headline leaderboard claims are computed through it, and no sensitivity analysis shows the '23/26 below 60' or model rankings survive reasonable alternative penalties.","rationale":"The paper's central assertion is about model reliability on length instruction following. The strongest independent evidence is the raw output-length behavior and the At Most/At Least tables, which support the qualitative finding that models struggle with long constraints. However, the headline '23/26 below 60' and the specific leaderboard positions are not raw measurements; they are values of an arbitrary exponential scoring function with hand-chosen constants. Some bounded metric is needed because Length Deviation suffers cancellation and outlier sensitivity, but the absence of any robustness analysis for k1=5 and k2=2 leaves the quantitative ranking vulnerable. This is exactly the reader's weakest assumption, and I agree with it. I did not find a more serious internal inconsistency: word counting is explicitly defined, dataset construction is documented, and the vendor-claimed-length experiments are supplemental. Therefore the existing CONDITIONAL verdict already captures the appropriate level of confidence; adding the proposed sensitivity analysis would resolve the concern, but no verdict change is needed.","tokens_in":65938,"tokens_out":5660,"duration_ms":49958,"concrete_test":"Recompute Table 1's Equal-To Length Score for all 26 models using alternative penalty constants (k1,k2) in {(3,3),(2,5),(10,2),(5,5)} and a symmetric log-ratio score 100*exp(-3|ln(L_out/L_con)|); then recompute the model ranking, the count of models below 60, and o3-mini's rank. If Spearman rank correlation with the published leaderboard stays ≥0.9 and the below-60 count stays ≥20 across all variants, the headline finding is robust; if mid-tier rankings swing or the count changes by more than 5 models, the quantitative leaderboard claims need to be qualified as metric-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All headline quantitative claims—23/26 models below 60 under Equal To, o3-mini top at 75.4, reasoning models beating long-text models—are computed from Length Score (Eq. 3) with hand-set k1=5, k2=2. The paper justifies k1>k2 by saying over-generation is more acceptable, but it provides no robustness analysis. This is load-bearing because LS is not an ordinal-preserving transform of LD once averaged across samples: changing k1/k2 changes the relative penalty on under- vs over-generation and can reorder models with different deviation profiles. For example, at |LD|=0.2, the current curve scores under-generation 36.8 vs over-generation 67.0; a symmetric or opposite asymmetry would substantially shift a model that systematically overshoots (e.g., Suri-I-ORPO, with mean LD 506%) relative to one that undershoots. The 'below 60' claim is likewise threshold-dependent: under the selected constants, a score of 60 corresponds to 10.2% under-generation or 25.5% over-generation—an arbitrary calibration. Independent support (raw word-count plots, At Most/At Least tables) suggests the qualitative conclusion that long constraints are hard is robust, but the specific leaderboard and the vendor-claimed-length gap rankings are not established without a sensitivity check.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LIFEBENCH, a benchmark for evaluating length instruction following in LLMs, with 10,800 instances across four NLG task categories, three control methods (Equal To, At Most, At Least), ten length constraints (16 to 8192 words), and English/Chinese data. The authors evaluate 26 models and report that most models fail on long length constraints, that almost all models fall short of their vendor-claimed maximum output lengths, and that reasoning models outperform long-text-specific models. The benchmark is released with code, data, and a dataset card. The central qualitative finding—that precise length following degrades sharply at longer constraints—is supported by raw output word counts and per-constraint tables, while the specific leaderboard numbers and the '23/26 below 60' claim depend on the hand-set penalty constants in the Length Score metric.","tokens_in":66197,"tokens_out":8372,"duration_ms":66170,"significance":"If the quantitative claims are made robust, LIFEBENCH would be a useful community resource: it covers a wider range of lengths and tasks than most prior instruction-following benchmarks, provides bilingual coverage, and includes diagnostic experiments (length awareness, input-length sensitivity, lazy-strategy prevalence) that go beyond simple pass/fail measurement. The paper ships code, data, and a dataset card, and its qualitative conclusion that long length instructions are unreliable is supported by multiple independent evidence streams, including mean output word counts and At Most/At Least success rates. The length-awareness analysis and the distinction between true generation limits and lazy refusal are valuable contributions.","major_comments":[{"comment":"The entire leaderboard and the headline claim that '23 out of 26 models score below 60' are computed through the Length Score with hand-set constants k1=5 and k2=2, but the paper provides no sensitivity analysis. Because LS is a nonlinear, asymmetric transform of LD, and because per-sample scores are averaged after the transform, different but equally reasonable (k1,k2) choices (e.g., symmetric k1=k2, or reversed asymmetry k1=2,k2=5) can reorder models with different over- vs. under-generation profiles and change the fraction of models below any threshold. This is load-bearing because o3-mini's top ranking and the 'below 60' statement are headline contributions. I ask the authors to report a grid sensitivity analysis over (k1,k2) (covering symmetric and reversed-asymmetry cases) and to state whether the main qualitative claims and the specific leaderboard order are stable.","section":"§4.3, Eq. (3); Table 1; Figure 1"},{"comment":"Model-level LS scores are reported as point estimates without confidence intervals or significance tests. Since the benchmark has only 360 unique content units, each repeated across 30 constraint/control combinations, the 10,800 instances are not independent; the effective sample size for model ranking is much smaller. Claims such as o3-mini (75.4) outperforming Claude-3.7-Sonnet-Thinking (61.3) and Gemini-2.5-Pro (60.0) need bootstrap CIs or per-model standard errors to be credible, particularly for models separated by small margins. The authors should add such uncertainty quantification to the leaderboard.","section":"§5.1, Table 1"},{"comment":"Empty outputs are excluded from the final results ('approximately 0.1% of instances with empty outputs per model'). Since empty generations and refusals are later characterized as a 'lazy strategy' failure mode (Section 5.3, Appendix H), their exclusion likely biases LS upward for the exact long-constraint cases where models fail. The fraction is small, but the authors should demonstrate robustness by either including empty outputs as LD=-100% in a re-analysis or reporting the exclusion rate per model and per constraint and showing that it does not affect any conclusion.","section":"§5.1, Evaluation Protocol"}],"minor_comments":[{"comment":"The 'Params' column lists Llama-3.1-70B-Instruct as '8B'; it should be '70B'.","section":"Table 1"},{"comment":"The API entries for Qwen3-32B-Thinking and Qwen3-235B-A22B-Thinking say 'deepseek-reasoner1', which appears to be a copy-paste error from the DeepSeek-R1 row.","section":"Table 10"},{"comment":"The headings 'Lable Task' should read 'Label Task'.","section":"Appendix C.1 and C.2"},{"comment":"There is a typo: 'legnth' should be 'length'.","section":"Appendix D.2.1"},{"comment":"Garbled Unicode sequences (e.g., '/uni00000014/...') appear in the rendered text of Figure 8 and elsewhere; the figures and captions should be regenerated with clean font embedding.","section":"Figure 8 and several figure captions"},{"comment":"The text says 'LongWriter-Llama3.1-8B exhibiting the largest Length Deviation reduction (↓ 13.6)', but Figure 6 plots Length Score; the metric name should be corrected to Length Score in that sentence.","section":"§5.3, paragraph on sensitivity to input length"},{"comment":"The section claims to 'demonstrate the rationality of LS and its hyperparameter settings', but it only justifies the choice qualitatively; adding the sensitivity analysis requested above would substantiate this claim.","section":"Appendix D.2.2"},{"comment":"In the data statistics paragraph, 'LENGTHBENCH' should be 'LIFEBENCH'.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical benchmark with good resource release, and the qualitative finding about long-constraint failure is well supported by raw word-count evidence. My main concern is the arbitrariness of the Length Score calibration: the leaderboard numbers and the 'below 60' claim are computed through hand-set k1=5, k2=2 with no robustness check. If the authors add a sensitivity analysis and uncertainty quantification, the paper would be acceptable; without those, the specific leaderboard claims are not established. The empty-output exclusion is a smaller but related robustness issue that should be addressed in the same pass."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Useful benchmark, not a paradigm shift. The dataset is the real contribution: 10,800 instances, bilingual, four task families, three control methods, ten constraints, 26 models, with code and data released. The central empirical finding — models comply at short lengths and degrade sharply beyond a few thousand words, and vendor-claimed maximum outputs are rarely reached — is believable and backed by raw word-count plots, not just the metric. The length-awareness and lazy-strategy analyses are nice add-ons.\n\nThe main soft spot is the scoring curve. Length Score in Eq. 3 uses hand-set k1=5, k2=2. The reasoning (over-generation is more acceptable) is fine, but there is no sensitivity analysis. The stress-test note is correct that LS is not an order-preserving transform of LD once averaged, so a different but equally defensible ratio could reorder models with different deviation profiles, e.g., heavy overshooters like Suri-I-ORPO versus undershooters. The '23/26 below 60' threshold is likewise calibration-dependent. Also no confidence intervals or repeated runs for stochastic API calls, and the ~0.1% empty outputs are excluded without much discussion. These are fixable and they do not overturn the qualitative conclusion, but they mean the leaderboard should be read as provisional until robustness is shown.\n\nOne smaller thing: the claim that reasoning models beat specialized long-text models is interesting but also LS-dependent; I'd want the raw distributions alongside LS when that gets cited.\n\nBottom line: send it to a serious venue. Ask for a sensitivity analysis over k1/k2, error bars or multiple seeds, and a clearer exclusion protocol. I'd bring it to my reading group and I'd cite it if I were working on length control.","headline":"Solid bilingual benchmark for length instruction following; qualitative findings hold, but the leaderboard rests on hand-set scoring constants that need a sensitivity check.","tokens_in":66747,"tokens_out":1781,"would_cite":true,"duration_ms":16140,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Almost no LLM reaches its advertised max output, 26-model test finds","keywords":["length instruction following","long-text generation","large language models","benchmark","length control","bilingual evaluation","output length awareness","reasoning models"],"falsifier":"Recompute the Equal To leaderboard with a symmetric penalty curve ($k_1=k_2=2$) and with a linear penalty $LS=100(1-|LD|)$; if the model ordering changes materially, or if the number of models scoring above 60 changes substantially, the paper's headline quantitative claim is an artifact of the chosen curve rather than a fact about the underlying outputs.","tokens_in":65731,"feed_emoji":"📏","tokens_out":10152,"duration_ms":76483,"temperature":0.7,"pith_summary":"LIFEBench is a benchmark designed to test one seemingly simple skill: whether large language models can produce output of a requested length, from 16 words to 8,192 words, in English and Chinese, across question answering, summarization, reasoning, and creative generation. The paper's central claim is that current models follow short length instructions acceptably but deteriorate sharply once targets grow, and that none can be relied on for exact-length outputs at long constraints. Its second claim is that almost all of the 26 tested models fall well short of the maximum output lengths their vendors advertise, despite long-context capabilities. A sympathetic reader would care because length control is not an abstract nicety: report drafting, summarization, and creative writing all depend on the model delivering what the prompt asked for. The paper also offers a diagnosis — models lack awareness of how many words they have actually produced, and sometimes quit or refuse early — and shows that reasoning models, which check length during their intermediate steps, currently do best but still fail at long targets.","feed_headline":"Almost no LLM reaches its advertised max output, 26-model test finds","feed_subtitle":"LIFEBench scores 10,800 bilingual prompts from 16 to 8,192 words; top model o3-mini hits 75.4 of 100.","key_machinery":"The load-bearing object is the Length Score (LS), an exponential penalty curve defined for each control method; for Equal To it is $LS = 100 e^{k_1 LD}$ when $LD<0$ and $LS = 100 e^{-k_2 LD}$ when $LD\\ge 0$, with $k_1=5$ and $k_2=2$, where $LD=(L_{\\text{output}}-L_{\\text{constraint}})/L_{\\text{constraint}}$ is the Length Deviation. The curve maps raw overshoot or undershoot to a bounded 0–100 score, punishing under-generation more harshly than over-generation, and every leaderboard number in the paper is computed through it. The supporting diagnostic machinery is the length-awareness experiment, in which models are asked to append their actual word count; the paper uses the gap between reported and true length to argue that word-count unawareness is a proximal cause of instruction failure.","core_discovery":"The paper establishes, on its own terms, that length instruction following is a measurable capability in which current LLMs are broadly deficient. On LIFEBench's 10,800 instances, 23 of 26 models score below 60 on the Equal To Length Score; for constraints of 4,096 words or more, no model consistently exceeds a score of 40. Under extended \"At Least\" prompts up to 32,768 words, only the Claude and Qwen families reach the requested length even in their longest decile, and most models stop far short of their vendor-claimed maximum output. The authors attribute this to three interacting causes: models cannot accurately self-report how long their own outputs are (reported counts anchor to the instructed length), long inputs further degrade compliance, and some models adopt lazy strategies such as premature termination or outright refusal. Reasoning models like o3-mini improve on this by drafting, counting, and revising during intermediate reasoning, reaching a Length Score of 75.4, yet the same mechanism still breaks down at the longest constraints.","pith_inferences":["My inference, not the paper's claim: the Length Score's hand-chosen asymmetry ($k_1=5$, $k_2=2$) means the leaderboard order is partly determined by the scoring curve; a symmetric curve would compress the gaps, so the 'most models below 60' headline needs a robustness check before it is treated as a stable fact.","My inference, not the paper's claim: the length-awareness result suggests a direct training signal — reward models for accurately self-reporting how many words they generated, or for placing a marker every 100 words — that could improve length following; the paper does not test this.","My inference, not the paper's claim: because the benchmark scores word count only, a model could game it by padding with repetitive or irrelevant text; combining the Length Score with a quality filter or a repetition detector would give a more complete picture of length instruction following.","My inference, not the paper's claim: the finding that Chinese prompts trigger over-generation hints at tokenization or pretraining-corpus biases; a natural next experiment is to see whether the effect disappears when output length is measured in tokens instead of characters."],"forward_implications":["If the paper's results hold, a user who asks for a 10,000-word output should expect a much shorter response or a refusal, and vendors' maximum output length claims should not be treated as achievable targets.","Long-context window size and length-instruction following appear to be separate capabilities; expanding context alone will not make models hit a requested length, so training and evaluation must target length control explicitly.","Length following is task- and language-dependent: summarization and Chinese prompts degrade scores substantially, so any practical system needs to set expectations per task and language rather than assume a single 'length control' skill.","Reasoning models' self-checking behavior (draft, count, revise) is currently the most effective mitigation, but it breaks down past a few thousand words, so inference-time verification alone is not a complete solution.","Benchmarks that measure only output quality and not length compliance are missing a failure mode that is visible in LIFEBench; length metrics should join quality metrics in standard LLM evaluation."],"supporting_citations":[{"why":"Supplies the prior length-constrained instruction following benchmarks (MT-Bench-LI, AlpacaEval-LI) whose short-range results LIFEBench extends.","marker":"[113]"},{"why":"Provides a model-agnostic length-control method and the 'Ruler' evaluation, the short-constraint baseline this paper contrasts with full-range evaluation.","marker":"[58]"},{"why":"Introduces LongWriter and its 10,000+ word generation claim, the main comparison point for whether long-text models can meet their advertised maximum output.","marker":"[8]"},{"why":"Supplies the Suri benchmark for multi-constraint long-form generation, a key comparison and source of long-text enhanced models evaluated here.","marker":"[83]"},{"why":"Documents premature termination in long generation, the failure pattern LIFEBench quantifies across models and target lengths.","marker":"[85]"},{"why":"Provides LongGenBench, a long-form generation benchmark in long-context LLMs, used to position LIFEBench's full-range length design.","marker":"[105]"},{"why":"Provides LongProc, a long procedural generation benchmark, another comparison point for the long-constraint part of the evaluation.","marker":"[111]"},{"why":"Supplies the reasoning taxonomy used to generate the benchmark's open-ended reasoning problems.","marker":"[44]"}],"fun_headline_variants":["LLMs flub length instructions beyond 4K words, LIFEBench finds","No LLM reaches its advertised max output, 26-model study","Reasoning models beat long-context for hitting word counts","Models stop short of claimed output lengths, 10,800 prompts show","Length instructions expose LLM limits: most fail past 4K words"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole leaderboard rests on one hand-picked scoring curve — the Length Score with $k_1=5$ and $k_2=2$ — and the paper does not show whether the model rankings, or the claim that most models score below 60, survive under a different but equally reasonable penalty curve.","fun_headline_variants_meta":{"raw":{"variants":["LLMs flub length instructions beyond 4K words, LIFEBench finds","No LLM reaches its advertised max output, 26-model study","Reasoning models beat long-context for hitting word counts","Models stop short of claimed output lengths, 10,800 prompts show","Length instructions expose LLM limits: most fail past 4K words"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00028,"raw_usage":{"total_tokens":1703,"prompt_tokens":1032,"completion_tokens":671,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":576}},"tokens_in":648,"tokens_out":671,"duration_ms":5860,"temperature":1.0,"reasoning_tokens":576,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:04:00.378607+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the Equal To leaderboard with a symmetric penalty curve ($k_1=k_2=2$) and with a linear penalty $LS=100(1-|LD|)$; if the model ordering changes materially, or if the number of models scoring above 60 changes substantially, the paper's headline quantitative claim is an artifact of the chosen curve rather than a fact about the underlying outputs.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Suri benchmark for multi-constraint long-form generation, a key comparison and source of long-text enhanced models evaluated here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides LongGenBench, a long-form generation benchmark in long-context LLMs, used to position LIFEBench's full-range length design."}],"review_version":1}