{"id":"5b953243-ecc6-4d4a-9919-52b94bd82de6","arxiv_id":"2501.08003","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A greedy entropy-maximizing sampler increases lexical diversity in a French corpus, but lexical and syntactic diversity correlate too erratically to use one as a proxy for the other.","lead":"This paper proposes a greedy algorithm that samples text to maximize lexical diversity, measured by entropy, and tests whether lexical diversity can stand in for the more expensive syntactic diversity. The algorithm raises lexical entropy beyond random sampling, but lexical and syntactic diversity correlate too erratically to use one as a proxy for the other.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Q2's 'correlations fluctuate' finding may be an artifact of the arbitrary 100K-sentence block size; document-level correlations should be checked.","rationale":"The reader's weakest assumption—parser prediction errors—is legitimate and is explicitly acknowledged in Section 7. However, the more load-bearing threat to the central Q2 claim is the arbitrary aggregation unit. Correlations computed on 100K-sentence blocks may not reflect the document-level relationship that the sampling heuristic actually exploits, and block size is a free parameter in the analysis. The paper itself concedes that another block size might yield different results, making this a directly acknowledged sensitivity gap. A sensitivity analysis across block sizes is straightforward and would settle whether the 'correlations fluctuate' conclusion is a property of the linguistic measures or of the chosen analysis granularity. This does not require rejecting the paper—the conclusions are already hedged, and the proposed check is feasible—so the reader's CONDITIONAL verdict remains appropriate. The parser-error concern is secondary but reinforces the need for caution; both point to the same practical implication: more validation is needed before the negative Q2 result is accepted as a general finding.","tokens_in":8150,"tokens_out":10449,"duration_ms":109505,"concrete_test":"Recompute the CLSD analysis on a subset of BASE (e.g., all 705 blocks or a 10% stratified sample) using block sizes of 1K, 10K, 50K, 200K, and 500K sentences, plus the individual-document level, with the same α grid (0–5) and the same parser output. If the fluctuation pattern, especially the union's negative dip near α≈1, does not persist across aggregation levels, then the Q2 conclusion is an artifact of the chosen block size and the manuscript must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Q2's central negative claim—that lexical diversity cannot serve as a reliable proxy for syntactic diversity—rests on CLSD computed over 705 blocks of 100K sentences each (Section 5). Section 7 concedes that the block size is arbitrary and that another value might change the results. Correlations over aggregate units are sensitive to the unit of aggregation (ecological correlation), and the sampling heuristic in Algorithm 1 operates on documents, not on 100K-sentence blocks. If the observed instability—including the union's negative Pearson dip near α≈1—is partly a by-product of mixing heterogeneous documents within blocks, varying block token counts, or Simpson's-paradox-like compositionality, then the experiment does not support the practical conclusion that lexical sampling cannot help construct syntactically diverse corpora. The parser-error risk (also flagged in Section 7) compounds this: rare syntactic phenomena are both the motivation for diversity sampling and the most error-prone, and parser errors are plausibly correlated with lexical frequency. Both threats are acknowledged, but no sensitivity analysis is reported; without controlling for aggregation level, the central Q2 result remains underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes lexical and syntactic diversity as Shannon and Rényi entropies computed over token forms and dependency subtrees, and proposes a greedy heuristic (Algorithm 1) for sampling a large French corpus so as to increase lexical diversity. The authors evaluate the heuristic (Q1) by comparing the entropy of the heuristic-selected corpus with random samples, and study (Q2) whether lexical diversity correlates with syntactic diversity across 705 blocks of 100K sentences from three subcorpora. They find a very large apparent increase over random for Q1 and report that lexical–syntactic correlations fluctuate with α, dataset, and aggregation, concluding that lexical diversity is not a reliable proxy for syntactic diversity.","tokens_in":8401,"tokens_out":2435,"duration_ms":27524,"significance":"The paper tackles a practically important question for large corpus construction: can cheap lexical diversity stand in for expensive syntactic diversity? Its formal framing via ecological diversity measures is appropriate, and the idea of varying the Rényi order α to interpolate between variety and balance is sound. The main empirical finding, if robust, would be a useful negative result: corpus builders should not assume that maximizing lexical diversity automatically yields syntactic diversity. The paper is also transparent in listing limitations. However, the two central empirical claims rest on experiments with testable flaws: the Q1 comparison uses an uncontrolled baseline, and the Q2 conclusion is based on a single arbitrary aggregation level with acknowledged parser noise but no sensitivity analysis. These issues are load-bearing, not cosmetic.","major_comments":[{"comment":"The 347σ claim is based on an uncontrolled baseline comparison. The heuristic output TOTAL is BASE plus HPLTdiverse, but the random baseline Nrandom is sampled from HPLTsmall alone. The entropy Hdiverse = 7.98 for TOTAL is therefore compared with Hrandom ≈ 7.656 for a corpus that does not contain BASE, so the difference conflates the contribution of BASE with the effect of the heuristic. To support the claim, the authors should compare TOTAL with BASE ∪ Nrandom (or, equivalently, compare the marginal increase H(TOTAL) − H(BASE) with H(BASE ∪ Nrandom) − H(BASE) over random seeds). Without this control, the reported significance level is not a valid test of the heuristic's effectiveness.","section":"§5, Q1"},{"comment":"The heuristic optimizes exactly the lexical entropy used for evaluation, so the increase in lexical diversity is partly self-fulfilling. A proper random-selection baseline would address this, but the current baseline does not, because of the issue in the previous comment. The claim that the heuristic 'significantly increases diversity relative to random sampling' would be more convincing if the authors reported the distribution of H(BASE ∪ Nrandom) and showed that Hdiverse lies far from it; the current 347σ figure does not establish this.","section":"§5, Q1 and §4"},{"comment":"The central negative finding—that lexical and syntactic diversity do not stably correlate—is computed over 705 blocks of 100K sentences each, and Section 7 concedes that the block size is arbitrary and that another value might change the results. Correlations over aggregate units are sensitive to the unit of aggregation (ecological correlation), and the sampling heuristic in Algorithm 1 operates on documents, not on 100K-sentence blocks. The observed instability, including the union's negative Pearson dip near α≈1, could be an artifact of mixing heterogeneous documents within blocks. The paper should report sensitivity to block size (e.g., document-level, sentence-level, and several intermediate sizes) or at least show that the qualitative pattern of correlations is stable. Without this, the conclusion that 'the use of lexical diversity did not increase syntactic diversity' is underdetermined.","section":"§5, Q2 and §7"},{"comment":"Section 7 states that the syntax used to compute syntactic diversity is parser-predicted and contains errors, and that rare phenomena may be badly predicted. Since rare syntactic phenomena are precisely the long-tail items that motivate diversity sampling, and since parser errors are plausibly correlated with lexical frequency, the Q2 correlations may reflect parser noise rather than genuine linguistic properties. The manuscript acknowledges this risk but presents no analysis of its magnitude. At minimum, the paper should quantify parser accuracy on the relevant rare constructions (e.g., by hand-checking a sample of rare dependency subtrees) or discuss why parser noise cannot explain the observed fluctuation pattern. This is a specific, testable concern that affects the validity of the central claim.","section":"§7 (Limitations) and §5, Q2"}],"minor_comments":[{"comment":"There are several typos and inconsistent notations: 'datsets' in the Introduction, 'subtress' in §2, 'HPLT small' vs 'HPLTsmall' in §4, and 'T OT AL' with irregular spacing in §4. The authors should also spell out the acronym CLSD when first used in §5.","section":"Throughout"},{"comment":"The exhaustivity array E is never instantiated with concrete values, and the target size S is described as 'here 3.1 billion' while earlier the goal is stated as 'around 1.5 billion' tokens similar to BigScience; this discrepancy should be clarified. The initial comparison H(W ∪ d) with d = ∅ (line 9) also needs a convention for the entropy of W ∪ ∅.","section":"Algorithm 1"},{"comment":"The formal definitions would benefit from stating explicitly that the entropy is computed on the empirical distribution of categories (maximum-likelihood estimates) and from giving the normalization step for token forms (footnote 3) more prominence, since it affects all later lexical diversity numbers.","section":"§2"},{"comment":"Figure 2 plots correlations for α from 0 to 5, but the paper does not report whether the trends converge for higher α or how the choice b = e influences the numbers; a sentence clarifying the role of the log base would help readers interpret the absolute entropy values.","section":"§5, Q2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practical question, and the formalization of diversity via Rényi entropy is a useful contribution. However, the two headline empirical results are not yet sufficiently supported: the Q1 comparison needs a properly controlled random baseline, and the Q2 conclusion needs robustness checks on aggregation level and parser noise. Both are fixable within the scope of the manuscript, but they are central rather than peripheral. I would be willing to review a revision that addresses these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: the paper's real value is its negative result on Q2 — lexical and syntactic diversity, measured as Rényi entropies, do not correlate stably across French corpora or α values. That is a useful caution for anyone planning to use cheap lexical diversity as a proxy for syntactic diversity. The authors are honest about the limits, and the ecological framing (Hill numbers, Rényi generalization) is a legitimate way to formalize diversity in NLP sampling.\n\nWhat's new: an application of the ecology diversity framework to corpus construction, a concrete greedy sampling heuristic (Algorithm 1), and the finding that CLSD fluctuates — including a negative Pearson dip near α=1 for the union of subcorpora. The heuristic itself is simple but clearly specified. No code or data are released, so reproduction depends on the description; that is a minor limitation but worth noting.\n\nThe main flaw is in Q1's evaluation. TOTAL (BASE plus heuristic-selected HPLT) is compared against random samples of HPLT alone, not against BASE plus random HPLT. Since BASE has entropy 7.02 and random HPLT alone is 7.656, the random baseline is missing a low-entropy component, which inflates the reported 347σ separation. The p-value text also contradicts its own footnote (the footnote says p is below double precision; the text says '≈ 0.5'). These are fixable with a rerun; the algorithmic idea may still hold.\n\nQ2's block size (100K sentences per block) is arbitrary, as the authors admit. The observed instability, especially the union's negative dip, could be an artifact of aggregating heterogeneous documents into coarse blocks. No sensitivity analysis is reported. Parser noise on rare constructions is another acknowledged threat, and it is plausible that parser errors correlate with lexical frequency. So the practical conclusion — that lexical sampling cannot help with syntactic diversity — is underdetermined, though the paper's own wording is appropriately cautious.\n\nWho gets value: researchers building corpora and working on diversity metrics; the Q2 cautionary result deserves attention even if the Q1 evaluation is reworked. I'd send this to a serious referee, but with a request for major revision: fix the baseline, add a block-size sensitivity check, and report variance across seeds for the heuristic itself.","headline":"Useful cautionary study of diversity measures, but the headline Q1 result rests on an uncontrolled baseline and the Q2 negative result is underdetermined by aggregation choices.","tokens_in":8885,"tokens_out":3063,"would_cite":false,"duration_ms":28868,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50","94A17"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes that a tractable entropy-guided sampling heuristic markedly increases lexical diversity, but that lexical diversity cannot reliably proxy syntactic diversity in French corpus construction.","keywords":["lexical diversity","syntactic diversity","data sampling","Rényi entropy","Shannon entropy","French corpus","dependency trees","corpus construction"],"falsifier":"Take the same 705 blocks, obtain manually validated gold parses for a random subset, and recompute the lexical–syntactic correlations; if correlations stabilize at high positive values once parser errors are removed, the claim that lexical diversity cannot proxy syntactic diversity would fail. Alternatively, find a dataset and $\\alpha$ where the correlation remains stably above 0.9 across resamples and genres.","tokens_in":7981,"feed_emoji":"📚","tokens_out":6813,"duration_ms":64472,"temperature":0.7,"pith_summary":"This paper asks how to build a large, diverse French corpus without paying for exhaustive search or full parsing. It proposes a greedy sampling algorithm that keeps documents whose addition raises Rényi entropy over word forms, and shows that the selected corpus reaches lexical entropy 7.98, far above 7.656 for random selection under the normal model (about 347 standard deviations). It then asks whether cheap lexical diversity can substitute for expensive syntactic diversity, where syntactic diversity counts whole dependency subtrees predicted by a parser. Across 705 blocks of an already-parsed French corpus, correlations between lexical and syntactic entropy fluctuate with the Rényi order $\\alpha$, with the corpus genre, and even with whether subcorpora are pooled, so the authors conclude that using lexical diversity does not increase syntactic diversity and that an arbitrary diversity measure may miss diversity-related properties. A sympathetic reader takes from this that corpus builders should parse at least a sample whenever syntactic diversity is the actual target.","feed_headline":"Lexical entropy cannot proxy syntactic diversity","feed_subtitle":"An entropy-guided sampler beats random selection for lexical diversity, but syntactic diversity still requires parsing.","key_machinery":"The central objects are formal diversity measures imported from ecology, applied to two category systems. For lexical diversity, categories are unique word forms and elements are token occurrences; for syntactic diversity, categories are complete dependency subtrees (POS labels plus dependency relations under the FTB-dep schema) and elements are subtree instances. The paper uses Shannon–Weaver entropy $H$ and its Rényi generalization $H_\\alpha$, where low $\\alpha$ emphasizes variety (the number of categories) and higher $\\alpha$ emphasizes balance of category frequencies. The sampling mechanism is Algorithm 1: starting from an initial corpus, scan candidate documents from a massive raw corpus, keep the candidate that most increases entropy when added, append it to the working corpus, and lower the exhaustivity level if the target size is not reached. The evaluation machinery is correlation analysis: compute lexical and syntactic $H_\\alpha$ on 705 blocks of 100K sentences and measure Spearman and Pearson correlations across $\\alpha$ values and across subcorpora.","core_discovery":"In the paper's own terms, the discovery is two-sided. For Q1, the answer is yes: the tractable heuristic, which scans candidate documents at decreasing exhaustivity levels and greedily appends the document that most increases $H_\\alpha$, produces a corpus with lexical entropy 7.98, while twenty random samples of the same target size follow a normal distribution with mean $\\approx 7.656$ and $\\sigma \\approx 9.027\\times 10^{-4}$; the heuristic result is about 347 standard deviations from the mean, so the gain is statistically overwhelming. For Q2, the answer is negative: correlation between lexical and syntactic $H_\\alpha$ (Pearson and Spearman) is not stable; it changes with $\\alpha$ (0 to 5), with dataset, and within genre, and pooling three subcorpora produces a negative Pearson correlation down to $-0.43$ near $\\alpha = 1$ even where individual subcorpora hover around zero. The authors state that the use of lexical diversity did not increase syntactic diversity, meaning a corpus sampled to maximize lexical entropy is not thereby syntactically diverse.","pith_inferences":["A practical design rule the authors gesture at: use a small parsed validation set to measure the lexical–syntactic correlation for the exact target genre and $\\alpha$ before deciding whether lexical sampling suffices; if the correlation is weak, parse directly.","The instability is consistent with lexical and syntactic diversity being driven by different statistical regimes—vocabulary growth and hapax distributions for the lexicon, structural recombination for syntax—so a single scalar entropy is unlikely to bridge them.","A testable extension would redo the correlation on manually corrected parses or a different dependency annotation scheme; if the instability persists, it is a property of French text rather than parser noise, which would generalize the paper's conclusion."],"forward_implications":["A corpus sampled by this greedy entropy heuristic will be lexically far more diverse than a random sample of equal size, with no exhaustive power-set search; the mechanism can be applied to new raw corpora.","Because lexical–syntactic correlations are unstable across datasets and $\\alpha$, a lexical-only sampling step cannot guarantee long-tail syntactic coverage; some parsing is unavoidable when syntactic diversity is the goal.","The Rényi order $\\alpha$ is not a free parameter to ignore: shifting from variety-oriented to balance-oriented diversity changes conclusions about how lexical and syntactic diversity relate.","Genre-specific validation is required: two parliamentary corpora show little resemblance in lexical–syntactic correlation, so a proxy tuned on one genre should not be transferred without checking.","Pooling heterogeneous data can produce correlation artifacts (e.g., $-0.43$ at $\\alpha \\approx 1$), so correlation estimates should be reported per source as well as for the union."],"supporting_citations":[{"why":"Defines the FTB-dep dependency annotation scheme, the category system used to compute syntactic diversity.","marker":"Candito et al. (2010)"},{"why":"Supplies the base entropy $H$ used as the diversity measure and the sampling objective.","marker":"Shannon and Weaver (1949)"},{"why":"Defines the $H_\\alpha$ family whose order parameter shifts between variety and balance and drives the correlation analysis.","marker":"Rényi (1961)"},{"why":"Provides the BigScience BASE corpus, the initial dataset whose entropy is 7.02 and on which the parsing and correlation experiments run.","marker":"Laurençon et al. (2022)"},{"why":"Provides HPLT, the massive raw French corpus from which the heuristic selects diverse documents.","marker":"De Gibert et al. (2024)"},{"why":"Supplies the formal diversity framework (elements, categories, variety, balance, disparity) that motivates defining lexical and syntactic diversity.","marker":"Morales et al. (2020)"},{"why":"Supplies the unifying diversity and evenness notation that justifies restricting the study to entropy measures.","marker":"Hill (1973)"}],"fun_headline_variants":["Lexical diversity gain, syntactic diversity lost","Entropy sampler boosts lexical, not syntactic, diversity","Lexical diversity does not buy syntactic diversity","Syntactic diversity can't be sampled via lexical proxy","Lexical sampling fails to diversify syntax"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Q2 conclusion rests on parser-predicted dependency subtrees being accurate enough that fluctuating correlations reflect real language properties; Section 7 concedes that predicted syntax contains errors, especially for rare phenomena.","fun_headline_variants_meta":{"raw":{"variants":["Lexical diversity gain, syntactic diversity lost","Entropy sampler boosts lexical, not syntactic, diversity","Lexical diversity does not buy syntactic diversity","Syntactic diversity can't be sampled via lexical proxy","Lexical sampling fails to diversify syntax"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1271,"prompt_tokens":837,"completion_tokens":434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":364}},"tokens_in":453,"tokens_out":434,"duration_ms":4149,"temperature":1.0,"reasoning_tokens":364,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:29:35.109469+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 705 blocks, obtain manually validated gold parses for a random subset, and recompute the lexical–syntactic correlations; if correlations stabilize at high positive values once parser errors are removed, the claim that lexical diversity cannot proxy syntactic diversity would fail. Alternatively, find a dataset and $\\alpha$ where the correlation remains stably above 0.9 across resamples and genres.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the FTB-dep dependency annotation scheme, the category system used to compute syntactic diversity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the base entropy $H$ used as the diversity measure and the sampling objective."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides HPLT, the massive raw French corpus from which the heuristic selects diverse documents."},{"cited_title":"Measuring Diversity in Heterogeneous Information Networks","cited_arxiv_id":"2001.01296","evidence_quote":"Supplies the formal diversity framework (elements, categories, variety, balance, disparity) that motivates defining lexical and syntactic diversity."}],"review_version":1}