{"id":"ba0340dc-7f20-4b00-a3ee-c39561dcfff5","arxiv_id":"2507.15092","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PATTR adds a target-length penalty to the Type-Token Ratio, producing a lexical diversity score with tunable, reduced short-text bias for LLM synthetic data.","lead":"The paper proposes PATTR, a modified Type-Token Ratio that adds a penalty when a response deviates from a chosen target length, aiming to remove the usual bias that favors shorter texts. The authors test it on a large corpus of LLM-written video scripts and report better or comparable diversity filtering than existing metrics.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PATTR's penalty can re-inject length bias in either direction depending on LT, and the headline filtering comparison is confounded: PATTR is scored without a length filter while MATTR/CR are length-binned, using length-sensitive evaluation metrics.","rationale":"The paper proposes a simple, transparent metric and provides a broad evaluation suite, which I read in good faith. The central condition for the claim to hold is that PATTR mitigates length bias without substituting a new bias, and that the reported filtering improvements come from lexical diversity rather than from length selection. The reader's weakest assumption concerned the need for a well-defined LT; that is real and acknowledged in the Limitations. My stress-test identifies a sharper, partly separate problem: even when LT is reasonable, the evaluation of the filtering claim is not length-matched. Figure 3 compares unfiltered PATTR against length-binned MATTR/CR, and the outcome metrics (ROUGE, BLEU, entropy) are themselves length-sensitive. The paper's own footnote about BLEU length bias in Table F.1 shows awareness of this effect, but the main comparison does not control for it. Table 3, which does control for length, shows only parity, so the 'consistently outperforms' claim is not yet established. In addition, the piecewise structure of Eq. (6) means PATTR is monotonically increasing with length below LT; this explains the +0.91 correlation in Table 1 and means the metric is not intrinsically length-agnostic. These concerns do not warrant rejection: the metric is clearly defined, the experiments are extensive, and the Limitations are honest about LT sensitivity. But the current evidence supports only a conditional acceptance, exactly as the reader concluded. I therefore leave the verdict unchanged, while adding a concrete length-matched test that would settle whether the filtering advantage is genuine.","tokens_in":17447,"tokens_out":6562,"duration_ms":69894,"concrete_test":"Re-run the top-k filtering evaluation in a fully length-matched design. For each model, bin all responses by length (e.g., 150-250, 250-350, 350-450, 450-550, 550-650). Within each bin, independently rank by PATTR (with LT at the bin midpoint), MATTR, and CR, then compute ROUGE-1/2/L, BLEU, and entropy on the top-10/100/1,000 selected sets in that bin. If PATTR does not beat or match MATTR/CR in the majority of bins and scenarios under these matched conditions, the headline filtering advantage is an artifact of length selection rather than of lexical diversity. A second smaller check: recompute Table 1 with LT set to the median response length of each model; if the absolute Spearman correlation remains large (e.g., >0.3) for several models, the claim of length-robustness fails even at a sensible LT.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (6) is piecewise: for L < LT, PATTR = |set(w)|/LT, so it increases with vocabulary size and therefore with length; for L > LT it behaves as U/(2L-LT). The penalty thus does not remove length dependence; it inverts it below LT. Table 1 confirms this: Spearman correlation with length is -0.42 at LT=100, near 0 at LT=275, and +0.91 at LT=400. Robustness is therefore entirely inherited from the user's choice of LT, and a misspecified LT actively re-injects the very bias the paper claims to mitigate.\n\nThe empirical validation has a matching confound. In Figures 3 and E.1, MATTR and CR are applied after restricting responses to length bins (200-600, 300-500, 350-450), while PATTR is evaluated on the unfiltered corpus (horizontal orange line). Because ROUGE, BLEU, and entropy are length-sensitive, selecting responses clustered near LT=400 can lower pairwise similarity scores even without a lexically better ranking. In the one comparison where all metrics are forced onto the same 350-450 word band (Table 3), PATTR is at parity or slightly below MATTR on 4/6-gram diversity, undercutting the claim of consistent superiority. The win-rate numbers in Tables 2/D.1 are also near-tautological for large LT: the penalty alone prevents selection of short responses, so low win rates do not independently demonstrate diversity.\n\nThus the central claim is not yet supported: PATTR is a length-penalized TTR whose bias depends on LT, and the evidence that it finds more diverse top-k sets is confounded by non-matched length distributions. The Limitations section acknowledges LT sensitivity, but not the asymmetry in the filtering comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Penalty-Adjusted Type-Token Ratio (PATTR), defined as PATTR(w, L_T) = |set(w)| / (|w| + ||w| - L_T|), where L_T is a user-specified target response length. On a synthetic corpus of video scripts generated by seven LLaMA, OLMo, and Phi instruct models, the authors compare PATTR with Moving-Average TTR (MATTR) and Compression Ratio (CR) in two settings: measuring bias toward short responses (win rates) and filtering top-10/100/1,000 responses for corpus diversity (ROUGE, BLEU, entropy, n-gram diversity, Wasserstein distance). The central claim is that PATTR is robust to prompt-induced length variations and yields on-par or better filtered diversity than MATTR and CR.","tokens_in":17748,"tokens_out":3393,"duration_ms":37945,"significance":"If the central claim were established, PATTR would be a simple, practical addition to the toolbox of lexical diversity metrics for synthetic-text curation, and the released corpus would be a useful resource. The paper has real strengths: the data generation is systematic (12,000 prompts per model, controlled variations in instructions, style, and topic), the evaluation is broad (five diversity measures, multiple model families, sensitivity analyses in Table 4 and Appendix F), and the limitations section is candid about the need to specify L_T and about the lack of human-judgment validation. However, the load-bearing claim of robustness is not supported by the evidence as presented: the sign of PATTR's length correlation changes with L_T (Table 1), and the headline filtering comparison is confounded by asymmetric length filtering between PATTR and the baseline metrics. These issues affect the main conclusions and require a substantive revision.","major_comments":[{"comment":"The claim that PATTR is 'robust to text length variations' (Abstract, §4.2) is not a property of the metric alone; it is inherited from the user's choice of L_T. Equation (6) is piecewise: for |w| < L_T the denominator equals L_T, so PATTR increases with vocabulary size and therefore with length; for |w| > L_T the denominator is 2|w| - L_T, so PATTR decreases with length. Table 1 confirms this: the Spearman correlation between PATTR and response length is -0.42 at L_T=100, near 0 at L_T=275, and +0.91 at L_T=400. Thus a misspecified L_T re-injects length bias in either direction. The paper should either reframe the contribution as 'a tunable length-penalized TTR that lets practitioners control length bias via L_T' or provide a principled procedure for setting L_T that does not depend on the corpus median. As written, the robustness claim is not established.","section":"§3.2, Eq. (6); §4.1, Table 1"},{"comment":"The main filtering comparison is confounded. In Figures 3 and E.1, MATTR and CR are applied only after restricting responses to length bins (0-2K, 200-600, 300-500, 350-450), while PATTR is evaluated on the unfiltered corpus (horizontal orange line). ROUGE, BLEU, and entropy are length-sensitive: selecting responses clustered near L_T=400 can lower pairwise similarity scores even if the ranking is not lexically better. To support the 'consistently outperforms' claim, PATTR must be compared with MATTR and CR under identical length constraints, with the length distributions of the selected sets reported for all metrics.","section":"§4.3, Fig. 3 and Fig. E.1"},{"comment":"When all three metrics are forced onto the same 350-450 word band, PATTR's 4-gram and 6-gram diversity (3.65 and 5.64) are below MATTR (3.66 and 5.65) and only slightly above CR (3.53 and 5.52). This directly undercuts the claim that PATTR provides consistently better or on-par diversity, and it suggests that PATTR's apparent advantage in Figure 3 is substantially driven by length selection rather than lexical ranking. The paper should report whether the differences are statistically significant and discuss this table more prominently, since it is the only matched-length comparison.","section":"§4.3, Evaluation-3, Table 3"},{"comment":"The win-rate results for PATTR at L_T=400 and L_T=600 are near-tautological. Since the penalty term ||w|-L_T| grows with deviation from L_T, and most responses in each model are shorter than 400 or 600 words (Table B.1), the metric by construction penalizes short responses. Low win rates for short sequences therefore do not independently demonstrate that PATTR identifies lexically diverse texts; they demonstrate that the penalty discourages short selections. A more informative control would compare PATTR against a baseline that applies the same length penalty to an otherwise length-neutral lexical score (e.g., TTR with the same denominator), or would report the lexical-diversity gain per unit of length difference.","section":"§4.2, Tables 2 and D.1"}],"minor_comments":[{"comment":"The Abstract states the corpus contains 'over 20M words,' while §3.1 says 'more than 50 million words (measured with whitespace-separated words).' These numbers should be reconciled.","section":"Abstract and §3.1"},{"comment":"The text says 'we use this set of prompts to generate video-scripts from 11 language models' but then lists seven models (OLMo-2 7B/13B, Llama-3.1 8B, Llama-3.2 1B/3B, Phi-3 Mini/Medium). The count '11' appears to be a typo.","section":"§3.1"},{"comment":"The table header includes a 'p-value' column, but the body reports only significance stars; either report actual p-values or change the column header.","section":"§4.1, Table 1"},{"comment":"The piecewise behavior of PATTR is not discussed explicitly. For |w| < L_T, PATTR = |set(w)| / L_T; for |w| > L_T, PATTR = |set(w)| / (2|w| - L_T). Stating this would make the length-dependence transparent and would help readers calibrate the metric's behavior.","section":"§3.2, Eq. (6)"},{"comment":"The caption says 'Except for ROUGE-1 with 0-2K filter, PATTR consistently outperforms MATTR and CR,' but the main text reports that PATTR outperforms in 14 of 16 scenarios. These statements should be aligned so the reader knows which scenarios are exceptions.","section":"§4.3, Fig. 3 caption"},{"comment":"For Phi-3-med, several ROUGE-1 and ROUGE-2 t-statistics are positive (e.g., +16.08 at L_T=400, W=128), and the surrounding text explains this as likely due to length bias in the metrics. Since this is an interpretation rather than a demonstrated fact, the paper should report the actual length distributions and, where possible, a length-matched comparison to support the claim.","section":"Appendix F, Table F.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's core idea is a simple length-penalized TTR, and the authors are transparent about the need to set L_T. In my view the paper is not ready for acceptance because the central robustness claim is contradicted by the metric's own length-correlation behavior (Table 1) and by the matched-length results in Table 3. However, the issues are fixable within the manuscript's scope: reframe the contribution as a controllable length-penalty family, run PATTR under the same length filters as MATTR/CR, and report the length distributions of all selected sets. I would not recommend rejection, because the proposed metric is clearly defined and the empirical effort is substantial. I also note that the in-text model count (11 vs. 7) and corpus-size figures should be corrected before the final version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, PATTR is a clean and simple idea: add |L - LT| to the denominator of TTR so that deviations from a task-specific target length are penalized. That is genuinely new relative to MATTR, MTLD, vocd, and the other length-correction heuristics in the related work. Second, the paper's central claim that PATTR is 'robust to length variations' is not actually supported by its own evidence. The metric does not remove length dependence; it swaps the sign of that dependence based on LT. Table 1 shows this directly: Spearman correlation with response length goes from -0.42 at LT=100 to +0.91 at LT=400. So robustness is inherited entirely from the user's choice of LT, a point the Limitations section does concede. The empirical sweep is broad and that should be credited. Seven models, more than 100k scripts (the abstract says 20M words, Section 3.1 says over 50M - an inconsistency worth fixing), and several external diversity measures. The sensitivity analysis in Table 4 also shows that with a sensible LT, PATTR can select longer and more diverse responses on ROUGE-L. There is real potential here for synthetic-data practitioners. The soft spots are proportionate to the overclaim. The filtering comparison in Figures 3 and E.1 is confounded: MATTR and CR are evaluated after length-binning the corpus, while PATTR is evaluated without any length filter. Since ROUGE, BLEU, and entropy are length-sensitive, this can make PATTR look better simply because its selected set has more length spread, not because it is lexically better. When the comparison is forced onto a common 350-450 word band (Table 3), PATTR is at parity or slightly below MATTR on 4/6-gram diversity - undercutting the 'consistently outperforms' claim. The win-rate results in Table 2 are also near-tautological for large LT: a metric that heavily penalizes short responses will naturally avoid ranking short responses first, so the low win rates do not independently demonstrate diversity. There are also smaller issues: no released code or data, despite the dataset being a stated contribution; no error bars in the main figures; and an internal inconsistency where the text says 11 models but lists only 7. None of this is fatal. The metric is well-motivated, the evaluation strategy is fixable, and the core idea is transparent enough to be useful. A serious referee could push for a matched-length headline comparison, a systematic LT sensitivity analysis, and released artifacts. I would engage with it: send to peer review, expect conditional acceptance after revision, and possibly bring it up in a reading group to discuss evaluation design for length-sensitive metrics. For my own work, I probably wouldn't cite it until the empirical case is tightened.","headline":"PATTR is a straightforward, interpretable length-penalized TTR worth knowing, but the empirical case for its superiority is weakened by a mismatched filtering comparison and by length bias that depends heavily on LT.","tokens_in":820,"tokens_out":1779,"would_cite":false,"duration_ms":56315,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Penalty-Adjusted Type-Token Ratio (PATTR) folds a task's target response length into the diversity score and, on synthetic LLM video scripts, selects more diverse outputs than MATTR or compression ratio while suppressing the usual…","keywords":["lexical diversity","type-token ratio","length bias","synthetic text","data filtering","prompt engineering","PATTR"],"falsifier":"Take a corpus with human-annotated lexical diversity and a known target length. If, with $L_T$ set correctly, PATTR's top-100 selections do not beat MATTR and CR on pairwise-similarity and entropy measures under equal length constraints, the central filtering claim fails. A sharper check: on a pool where all responses share the true target length but lengths vary, PATTR's rank order should stay stable as lengths deviate; if changing $L_T$ by a small amount reverses which responses rank top, the metric is not robust to the parameter it depends on.","tokens_in":1827,"feed_emoji":"📊","tokens_out":2201,"duration_ms":77865,"temperature":0.7,"pith_summary":"Conventional lexical diversity metrics such as Type-Token Ratio, Moving-Average TTR, and Compression Ratio systematically reward short texts, so when prompt changes alter response length they rank the least informative outputs as the most diverse. This paper argues that the bias can be corrected by folding the task's target response length directly into the metric: Penalty-Adjusted Type-Token Ratio (PATTR) divides the number of unique words by the word count plus the absolute deviation from the target length. On a large synthetic corpus of video scripts generated from seven instruction-tuned language models, the authors show that PATTR's correlation with length can be tuned from negative to near-zero to positive by choosing $L_T$, and that top-10/100/1,000 selections made with PATTR are on par or better by ROUGE, BLEU, entropy, n-gram diversity, and Wasserstein distance than selections made with MATTR or CR. If the claim holds, practitioners filtering synthetic data for model training can choose diversity scores that respect the length a task actually requires instead of inheriting a hidden preference for short text.","feed_headline":"PATTR removes length bias from LLM text diversity scores","feed_subtitle":"Penalizing responses that miss a task's target length yields more diverse selections than standard metrics.","key_machinery":"The load-bearing object is the penalty-adjusted denominator $|w| + \\|\\,|w|-L_T\\,\\|_1$, where $L_T$ is a user-specified target length for the task. Replacing the plain TTR denominator with this term makes the score depend on absolute distance from the target rather than on raw length, so the bias toward short responses is controlled by one tunable number. The paper's empirical machinery is a structured prompting setup: a fixed task description, incrementally added formatting instructions, varied style inputs, and varied user topics generate 12,000 responses per model, making response length and diversity observable under controlled prompt perturbation. The metric is evaluated by ranking and filtering responses, then measuring homogenization with pairwise similarity and distributional diversity.","core_discovery":"The central discovery is that a single penalty term $\\|\\,|w|-L_T\\,\\|_1$ in the denominator of TTR turns a length-biased score into a length-aware one. PATTR is defined as $\\mathrm{PATTR}(w, L_T) = |\\mathrm{set}(w)| / (|w| + \\|\\,|w| - L_T\\,\\|_1)$, where $L_T$ is the task's target response length. Shorter-than-target and longer-than-target responses are penalized symmetrically, so the metric no longer automatically crowns the shortest script; the paper shows empirically that varying $L_T$ from 100 to 275 to 400 flips PATTR's Spearman correlation with response length from $-0.42$ to $+0.03$ to $+0.91$. In filtering experiments, PATTR with $L_T=400$ selects top-10/100/1,000 video scripts that have lower pairwise ROUGE and BLEU similarity, higher per-token entropy, higher n-gram diversity, and a smaller Wasserstein distance to a uniform vocabulary distribution than selections made with MATTR or compression ratio, while staying closer to the target length. The authors also show PATTR's selected outputs are both longer and significantly more diverse than MATTR's, which they take as evidence that the improvement is not a shortcut through short-text bias.","pith_inferences":["A testable extension is to use PATTR as the filter inside synthetic-data distillation loops, checking whether training on PATTR-filtered data slows the diversity collapse that repeated self-training induces.","The same penalty idea could be transferred to syntactic and semantic diversity measures; if those metrics also have a length component, a deviation penalty of the same form may remove bias without re-tuning.","For open-ended tasks where no natural $L_T$ exists, one could set $L_T$ from the model family's own length distribution, which would make PATTR a normalization rather than a task constraint; the paper does not explore this.","The direction of the bias is not fixed: the paper's correlation results imply a user could deliberately set $L_T$ low or high to favor short or long responses, which may be undesirable in evaluation but useful for controlled data augmentation."],"forward_implications":["Data curation pipelines that filter synthetic text by diversity can set $L_T$ to the desired response length, so short filler and overlong rambling outputs are both downweighted instead of the short ones being automatically rewarded.","Analyses of prompt engineering effects on diversity become interpretable: a change in measured diversity can be separated from a change in length once length bias is explicitly parameterized.","For tasks with known length constraints, such as essay or short-story generation, PATTR offers a per-response ranking that does not need pairwise corpus comparisons, so it scales linearly with the number of strings.","Because PATTR remains a per-response score, it can rank large synthetic corpora in linear time, unlike pairwise similarity methods whose cost grows quadratically with corpus size."],"supporting_citations":[{"why":"Introduces Moving-Average TTR, the main per-response baseline whose length bias PATTR is designed to fix.","marker":"Covington and McFall, 2010a"},{"why":"Documents the length dependence of lexical diversity metrics, motivating the penalty term.","marker":"McCarthy and Jarvis, 2010"},{"why":"Supplies the Compression Ratio metric and n-gram diversity evaluation, and provides evidence of length correlation in diversity scores.","marker":"Shaib et al., 2024a"},{"why":"Provides ROUGE, used as a pairwise homogenization score for evaluating filtered corpora.","marker":"Lin, 2004"},{"why":"Provides BLEU, used as a second pairwise homogenization score for filtering evaluations.","marker":"Papineni et al., 2002"},{"why":"Provides the Wasserstein distance used to compare selected vocabularies against a uniform distribution.","marker":"Vaserstein, 1969"},{"why":"Supplies the small language models used to compute average per-token entropy of filtered corpora.","marker":"Allal et al., 2025"}],"fun_headline_variants":["Penalty term fixes length bias in text diversity metrics","PATTR: length-aware diversity metric outshines TTR","One penalty flips length bias in LLM text scoring","Target length penalty fixes TTR short-text bias","Better diversity picks with penalty-adjusted TTR"],"cache_read_input_tokens":20352,"weakest_assumption_plain":"The whole correction rests on choosing a target response length $L_T$ that is genuinely appropriate for the task and fair across all models and prompts; if that number is wrong, the penalty simply re-injects length bias in the opposite or an unintended direction.","fun_headline_variants_meta":{"raw":{"variants":["Penalty term fixes length bias in text diversity metrics","PATTR: length-aware diversity metric outshines TTR","One penalty flips length bias in LLM text scoring","Target length penalty fixes TTR short-text bias","Better diversity picks with penalty-adjusted TTR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1471,"prompt_tokens":1072,"completion_tokens":399,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":323}},"tokens_in":688,"tokens_out":399,"duration_ms":4591,"temperature":1.0,"reasoning_tokens":323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:40:57.301652+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a corpus with human-annotated lexical diversity and a known target length. If, with $L_T$ set correctly, PATTR's top-100 selections do not beat MATTR and CR on pairwise-similarity and entropy measures under equal length constraints, the central filtering claim fails. A sharper check: on a pool where all responses share the true target length but lengths vary, PATTR's rank order should stay stable as lengths deviate; if changing $L_T$ by a small amount reverses which responses rank top, the metric is not robust to the parameter it depends on.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the length dependence of lexical diversity metrics, motivating the penalty term."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Wasserstein distance used to compare selected vocabularies against a uniform distribution."}],"review_version":1}