{"id":"2b93645a-9e41-43c2-b3f7-e35920132fee","arxiv_id":"2412.17321","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A compression-based distance between original and edited AI text correlates with human editing time, but the metric is a known compression distance applied to a new task.","lead":"This paper tests a zip-compression trick for estimating how much human work went into editing an AI-written text, and finds it tracks real editing time better than standard similarity scores in its experiments. It also reports that AI models seem to edit less when told to edit quickly, which the authors interpret as aligning with the metric.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's pooled Pearson correlations do not control for source-text length or annotator clustering; both edit time and the LZ-based distance scale with text length, so the reported 0.81 advantage over Levenshtein may be an artifact.","rationale":"The strongest empirical claim is that the proposed metric correlates with human editing effort better than alternatives (Table 1, Section 5). For that claim to hold, the reported Pearson correlations must reflect the metric's ability to track editing effort rather than shared nuisance variation. The data structure violates the independence assumption: 800 observations come from 200 source texts each edited by four annotators, and source length varies widely. Both edit time and the LZ-based distance are length-sensitive: longer texts require more reading and editing time, and LZ factorization phrase counts grow with text length. Without controlling for length or clustering, the pooled correlations are not interpretable as evidence for the metric's validity, and the W/Knowledge manipulation changes source length asymmetrically, potentially favoring the compression metric for reasons unrelated to editing effort. This is the same weakness the reader identified, and I agree it is load-bearing. The concern is testable with partial correlations and cluster-robust or mixed-model analysis; if those analyses preserve the advantage, the central claim would be substantially strengthened. I do not see an independent internal inconsistency that would justify moving to REJECT, so the appropriate outcome remains CONDITIONAL as the reader stated.","tokens_in":10210,"tokens_out":5750,"duration_ms":63221,"concrete_test":"Fit mixed models of the form edit_time ~ metric_value + source_length + (1|source_id) + (1|annotator_id) separately for compression distance and Levenshtein using the W/Knowledge condition; also compute the partial correlation of each metric with edit_time after controlling for source and target length. Compare the two metric coefficients using a cluster bootstrap resampled by source_id (200 clusters). If the compression-distance coefficient or partial correlation is not significantly larger than Levenshtein's, the Table 1 advantage is a length/clustering artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2's headline result (Table 1: compression r=0.81 vs Levenshtein r=0.59, with knowledge concatenated) is computed as a pooled Pearson correlation over 800 observations that are not independent: the same 200 source texts are edited by four annotators (Section 4.1.1). Neither edit time nor d(S->T)=LZ(S|T)-LZ(S) is adjusted for text length, and source length varies substantially (mean about 1008, SD about 260, Fig. 2). Edit time inevitably grows with the amount of text read and modified; LZ phrase counts also grow with |S|+|T|. Hence a high pooled r can arise solely from a shared length factor, without any real per-edit-effort signal. The W/Knowledge condition appends the expert knowledge K to the source only, injecting an additional source-length component into every metric; compression distance is especially sensitive to long repeated substrings, so this setup can differentially inflate its correlation relative to Levenshtein. No partial correlation, within-source centering, mixed model, or cluster-robust significance test is reported, so the claim that compression distance outperforms baselines is not supported by the presented statistics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a compression-based edit distance, d(S→T) = LZ(S|T) − LZ(S) (Section 3), grounded in LZ77 factorization, intended to measure the human effort of post-editing LLM-generated texts. The authors contribute a new dataset of 200 accounting questions with human edits by four expert annotators and synthetic LLM edits under three instructed scenarios, and they evaluate their metric against Levenshtein, TER, CharacTER, BLEU, ROUGE-L, METEOR, and BERTScore on this dataset and on IWSLT2019. The headline result is a Pearson correlation of 0.81 with human edit time when the expert knowledge text is concatenated to the source, compared with 0.59 for Levenshtein under the same condition. The paper also reports KNN regression results and synthetic-edit analyses intended to show that the metric tracks editing intensity and that LLMs' notion of editing speed aligns with the distance.","tokens_in":10344,"tokens_out":3347,"duration_ms":33846,"significance":"If the empirical claims are supported by sound statistics, this is a valuable contribution: the metric is parameter-free, has a clear information-theoretic interpretation, and runs in linear time via the Crochemore et al. algorithm, which is a practical advantage over BERTScore and other expensive metrics. The authors provide code and data, and the metric definition is transparent and reproducible. However, the central claim of superiority over existing metrics is currently under-supported because the key comparison (Table 1) uses an analysis that does not account for clustering or text-length confounding, and the reported advantage depends on a particular concatenation protocol that can differentially favor compression-based measures. The IWSLT2019 results actually show compression distance is roughly on par with Levenshtein, so the only place where compression distance clearly outperforms baselines is the analysis most vulnerable to confounding.","major_comments":[{"comment":"The Pearson correlations in Table 1 are computed on the pooled 800 observations (4 annotators × 200 source texts) as if they were independent, but the observations are clustered by source text: each of the 200 generated answers is edited by all four annotators, so observations sharing a source text are correlated. Both edit time and the compression distance d(S→T) scale with text length, and source length varies substantially (mean 1008, SD 260, Figure 2). A pooled Pearson correlation can therefore be inflated by the shared length factor even if the metric has no per-edit-effort signal. The paper does not report partial correlations controlling for source length, within-source centering, mixed-effects models, or cluster-robust significance tests. The claim that compression distance outperforms Levenshtein (0.81 vs. 0.59) is not supported by the presented statistics. Please re-analyze with hierarchical or cluster-robust methods, or provide per-source averaged correlations, and report confidence intervals.","section":"Section 4.2, Table 1"},{"comment":"The comparison labeled \"W/ Knowledge\" concatenates the expert knowledge text K to the source S only, leaving the target T as the edited answer. This asymmetric construction is likely to favor compression distance disproportionately, because LZ-based measures are specifically sensitive to long repeated substrings and can detect copied knowledge fragments in T, while Levenshtein and other character/word-level metrics do not exploit such block-level repetitions to the same degree. The paper interprets the improved correlation as evidence that human editors reuse knowledge text, but the protocol itself may be the source of the improvement. To support the claim, the authors should either (i) justify the concatenation protocol on independent grounds and compare it against concatenating K to both S and T, or (ii) report the W/o Knowledge result as the primary comparison and treat the W/Knowledge result as a secondary, sensitivity analysis. As it stands, the key advantage in Table 1 is concentrated in a condition that is confounded with the metric's design.","section":"Section 4.2, W/Knowledge condition"},{"comment":"On the IWSLT2019 dataset, where correlations are reported per annotator (avoiding the clustering issue), compression distance does not outperform Levenshtein: for edit-time correlations, the ranges are 0.55–0.73 for compression distance and 0.55–0.73 for Levenshtein, with Levenshtein winning on A1 and A4 and compression distance winning on A0, A2, and A3. The paper attributes the parity to the absence of block operations in the PET tool; however, this substantially weakens the general claim that compression distance 'outperforms' existing metrics. The authors should either temper the claim or provide a more direct test of block-operation sensitivity, e.g., by introducing controlled block edits into a subset of the IWSLT2019 data and showing a differential gain.","section":"Section 4.2, Table 2"},{"comment":"The synthetic-edits analysis (Figure 3) demonstrates that the compression distance is smaller in the 'fast' and 'similar' scenarios than in the 'normal' scenario, which is consistent with the authors' expectations. However, no baseline metric is evaluated on the same synthetic data. It is unclear whether this ordering is specific to compression distance or would also be observed with Levenshtein, TER, or even simple length difference, since the scenarios are defined by instructions that likely reduce the amount of textual change. Without a baseline comparison, the claim that 'LLMs exhibit an implicit understanding of editing speed that aligns well with our metric' (Abstract) is not established. Please add baseline metric values and, if possible, a statistical test (e.g., paired comparison across questions) for the scenario-ordering effect.","section":"Section 4.2, Synthetic Edits"},{"comment":"Tables 3 and 4 report KNN regression R² values on random 80/20 splits of the data. For the human-edited dataset (Table 4), the 800 observations again are not independent because of the 4-annotator-per-source design; a random split can place the same source text in both training and test sets, inflating the KNN R² for all metrics, especially those that are sensitive to text length. This may explain the very high R² (0.6316) for compression distance. Please evaluate using a grouped split (e.g., all annotations of a source in the same fold) or report per-source cross-validation, and also report R² for a length-only baseline to check for confounding.","section":"Section 4.2, KNN regression"}],"minor_comments":[{"comment":"The definition of d(S→T) uses the concatenation 'S | T' but the delimiter is not specified; the choice of delimiter can affect LZ77 factorization and should be stated explicitly, along with whether the delimiter is a special character or a whitespace token.","section":"Section 3"},{"comment":"The text says 'All reported correlation coefficients are statistically significant (p <0.05)', but no test procedure, degrees of freedom, or correction for multiple comparisons is described; please specify the significance test and whether cluster-robust p-values were used.","section":"Section 4.2"},{"comment":"In the paragraph beginning 'From Table 1', there is a typo: 'suggesting thatn-gram overlap' should be 'suggesting that n-gram overlap'.","section":"Section 5"},{"comment":"The sentence 'We apply the algorithm proposed by Crochemore et al. (2008) for computing the Lempel-Ziv factorization in linear time from suffix arrays. as implemented in pydivsufsort2' begins with a capital 'We' but the second sentence starts with 'as', which is a sentence fragment and missing a period; please correct the punctuation.","section":"Section 3"},{"comment":"Figure 1 is referenced in the text before it appears in the provided PDF; please ensure the figure placement is consistent in the final version.","section":"Section 4.1.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript describes an interesting and potentially practical metric, and the release of code and data is commendable. My main concern, which I have conveyed to the authors, is that the evaluation does not currently support the paper's central claim of superiority over existing metrics. The only analysis in which compression distance clearly outperforms the baselines (Table 1, W/Knowledge) is also the one with the most severe statistical problems (pooled non-independent observations and an asymmetric concatenation protocol). The IWSLT2019 results, which are not subject to the same concerns, show only parity with Levenshtein. I would be willing to reconsider a revised version that addresses these statistical issues and, in particular, provides a clustered analysis and a length control. The paper is within the scope of cs.CL and the metric could be a useful contribution if the evidence is strengthened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is an application paper, not a new metric. The distance in Eq. (1) is exactly the conditional LZ distance from Ergün et al. (2003), which the authors cite, so the \"novel compression-based edit distance\" headline is wrong. What is new is the dataset: 200 French accounting Q&A answers generated by Claude, 800 timed human edits by four annotators, 600 synthetic edits, plus a run on IWSLT2019. Code and data are public. That is genuinely useful for people building LLM evaluation pipelines.\n\nThe empirical story is suggestive but the headline result is not supported as presented. Table 1 pools 800 observations that are not independent: four annotators edit the same 200 source texts, and both edit time and LZ phrase counts scale with text length (Fig. 2: source mean ~1008, SD ~260). The reported r=0.81 vs Levenshtein r=0.59 in the W/Knowledge condition could arise from a shared length factor alone. The W/Knowledge condition appends the expert knowledge to the source only, adding a large block of repeated material that the compression distance is especially sensitive to; this differentially helps compression over Levenshtein. No partial correlations, no cluster-robust significance, no test of whether the difference between correlations is significant.\n\nThe IWSLT2019 results (Table 2) are more encouraging and more credible: per-annotator correlations show compression and Levenshtein essentially tied, which is an honest result. The KNN regressions on the proprietary data (Table 4) show a big gap (R2 0.63 vs 0.33), but they use the same possibly-confounded data. The synthetic \"fast edit\" result is close to tautological: you prompt the LLM to take less time and it makes fewer changes; that validates the prompt, not the metric.\n\nWhat should happen: the authors should acknowledge the prior definition, replace the pooled test with a mixed model or cluster-robust test that controls for source length (e.g., length as covariate or within-source centering), and show the W/Knowledge result is robust to also appending knowledge to the target or using a symmetric condition. If the advantage over Levenshtein survives that, the paper becomes a solid empirical contribution. If not, it still provides a useful dataset.\n\nFor peer review: yes, send it out. The question matters, the data are public, and the flaws are fixable with statistics, not architecture. I would not cite this in my own next-12-month work because the metric is prior art and the evaluation is not yet trustworthy, but I would keep an eye on the dataset. Reading group: maybe, as a case study in how length confounds can inflate metric correlations.","headline":"Useful dataset, overclaimed novelty, and a statistics problem: the headline result is likely inflated by a length confound and the metric is prior art, but the paper deserves a serious referee for its empirical contribution.","tokens_in":10948,"tokens_out":2763,"would_cite":false,"duration_ms":26918,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Counting LZ77 phrase differences between an original LLM answer and its edited version correlates with human editing time at up to r = 0.81, exceeding standard automatic metrics.","keywords":["compression distance","LZ77","edit distance","post-editing effort","LLM evaluation","human-in-the-loop","segment rearrangement","text compression"],"falsifier":"Recompute the correlation between compression distance and edit time after partialling out source-text length (or fitting a mixed-effects model with a random intercept for each of the 200 source texts). If the partial correlation falls to near zero, the claim that the metric tracks editing effort collapses. A second check is to run the same correlation on a held-out corpus where editors are forbidden from copying the reference knowledge text; the metric's advantage over Levenshtein should come from its handling of substring reuse, so this test isolates whether that advantage is genuine.","tokens_in":9918,"feed_emoji":"📏","tokens_out":10865,"duration_ms":87762,"temperature":0.7,"pith_summary":"The paper's central claim is that the amount of human effort spent editing an LLM-generated text can be measured as a difference in LZ77 compression phrase counts between the original and the edited text. This is the compression-based edit distance $d(S\\to T)=\\mathrm{LZ}(S\\mid T)-\\mathrm{LZ}(S)$, and the authors argue it captures the block moves, deletions, and copy-paste reuse that dominate real post-editing but are missed by BLEU, ROUGE, TER, and BERTScore. They support the claim with a new dataset of 200 French accounting answers generated by an LLM, each edited by four expert annotators with recorded edit times, plus synthetic edits under normal, structure-preserving, and fast instructions. On this data the metric's Pearson correlation with edit time reaches 0.81 and a 5-nearest-neighbour regressor using only the metric predicts edit time with $R^2=0.63$, roughly double the best baseline. If the result is right, it gives a parameter-free, linear-time tool for estimating how much human work a model's output will require.","feed_headline":"Compression distance predicts how long humans edit LLM output","feed_subtitle":"Counting LZ77 phrase differences correlates with edit time at up to r=0.81, beating BLEU and BERTScore.","key_machinery":"The load-bearing object is the LZ77 factorization of a string, the greedy parsing into the longest substrings that have already occurred earlier in the text. The paper's metric is the difference $d(S\\to T)=\\mathrm{LZ}(S\\mid T)-\\mathrm{LZ}(S)$ in phrase counts, which measures how much new information the edited text contributes once the original has already been parsed. The theoretical justification is a constant-factor approximation result connecting this quantity to the minimum number of character-level and substring-level edit operations (moves, copies, deletions), and the practical engine is a linear-time algorithm that computes the factorization from suffix arrays. This machinery lets the distance treat a copied sentence from a reference text as cheap, while still charging for deletions, reorderings, and fresh content—behaviour that aligns with how human post-editors actually work.","core_discovery":"The paper establishes that post-editing effort is well approximated by a compression distance defined through the LZ77 parsing: $d(S\\to T)=\\mathrm{LZ}(S\\mid T)-\\mathrm{LZ}(S)$, where $\\mathrm{LZ}$ counts phrases in the greedy factorization. This quantity upper-bounds, within a constant factor of 4, the minimum number of character edits, substring relocations, duplications, and deletions required to transform the original into the edited text, and it can be computed in linear time using a suffix-array-based LZ factorization. Empirically, on the paper's human-edited accounting answers the distance correlates with measured edit time at $r=0.71$ without and $r=0.81$ with the expert knowledge text concatenated to the original, outperforming Levenshtein, CharacTER, TER, BLEU, ROUGE-L, METEOR, and BERTScore. On the IWSLT2019 post-editing benchmark the metric tracks keystrokes ($r\\approx 0.82$ to $0.89$ across five annotators) and edit time nearly as well as Levenshtein distance, and synthetic experiments show that edits produced under 'fast' or structure-preserving instructions have systematically lower compression distances. The authors conclude that the metric reflects both the mechanical and the cognitive load of post-editing, and that language models encode the notion of editing speed in a way that matches this compression-based measure.","pith_inferences":["If the length-confounding worry is resolved, the metric could be adopted outside LLM evaluation—for example, estimating revision effort in document management or translation workflows, where block operations and copy-paste dominate.","The sensitivity to substring reuse suggests a practical testable use: flagging edits that are pure copy-paste from a reference source as low-effort, which could be useful for quality control of outsourced editing.","A natural extension would be to check whether the 4-approximation's correlation with edit time persists when edits are performed by non-experts or in languages without the strong reference-knowledge setup used here, since the accountancy dataset has a very specific edit regime.","The metric's link to editing speed could give a principled way to calibrate LLM prompts: the 'fast edit' scenario systematically reducing compression distance implies the model internalizes effort not just as token count but in a way consistent with substring-level operations."],"forward_implications":["Post-editing effort on LLM output can be scored in linear time without any trained model or similarity embeddings, making it usable on very large document collections.","Because the metric charges little for reuse of a reference knowledge base, it agrees with the observation that editors often copy sentences from supplied sources; traditional metrics penalize that reuse and therefore mis-rank editing effort.","The synthetic experiments show that edits requested under 'similar' or 'fast' instructions yield lower compression distances, so the metric could be used to diagnose how deep an editing pass actually was.","On the IWSLT2019 benchmark, the compression distance is competitive with Levenshtein distance for keystroke and time prediction and clearly better than TER, BLEU, METEOR, and BERTScore, which suggests the advantage is not specific to the new dataset.","Combining the metric with others in a KNN regressor did not improve prediction beyond the metric alone, suggesting the compression distance already captures most of the editing-effort signal."],"supporting_citations":[{"why":"Defines the segment rearrangement distance and proves the constant-factor approximation that the metric relies on.","marker":"(Ergün et al., 2003)"},{"why":"Gives the linear-time LZ factorization algorithm from suffix arrays used in the implementation.","marker":"(Crochemore et al., 2008)"},{"why":"Provides the IWSLT2019 post-editing dataset with keystrokes and edit times used for validation.","marker":"(Scarton et al., 2019)"},{"why":"BLEU is a baseline n-gram overlap metric compared against the compression distance.","marker":"(Papineni et al., 2002)"},{"why":"TER is a baseline post-editing effort metric that the compression distance is compared against.","marker":"(Snover et al., 2006)"},{"why":"ROUGE-L is a baseline summarization metric used in the comparison.","marker":"(Lin, 2004)"},{"why":"BERTScore is the semantic-similarity baseline used in the comparison.","marker":"(Zhang et al., 2020)"},{"why":"Character-level edit distance is the classical baseline most comparable to the compression distance.","marker":"(Levenshtein, 1966)"}],"fun_headline_variants":["LZ77 edit distance beats BLEU at predicting human edit effort","Compression metric estimates post-editing time better than standard scores","New edit metric uses LZ77 to gauge human effort on LLM text","Compression-based edit distance rivals BLEU for LLM post-editing","LZ77 distance predicts human edit time on LLM texts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the strong correlation between compression distance and edit time is not an artifact of both quantities growing with the length of the source text—the same 200 texts were edited by all four annotators, yet the observations are treated as independent in the reported Pearson correlations.","fun_headline_variants_meta":{"raw":{"variants":["LZ77 edit distance beats BLEU at predicting human edit effort","Compression metric estimates post-editing time better than standard scores","New edit metric uses LZ77 to gauge human effort on LLM text","Compression-based edit distance rivals BLEU for LLM post-editing","LZ77 distance predicts human edit time on LLM texts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000649,"raw_usage":{"total_tokens":3038,"prompt_tokens":1062,"completion_tokens":1976,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":678,"completion_tokens_details":{"reasoning_tokens":1882}},"tokens_in":678,"tokens_out":1976,"duration_ms":12496,"temperature":1.0,"reasoning_tokens":1882,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:35:09.898973+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the correlation between compression distance and edit time after partialling out source-text length (or fitting a mixed-effects model with a random intercept for each of the 200 source texts). If the partial correlation falls to near zero, the claim that the metric tracks editing effort collapses. A second check is to run the same correlation on a held-out corpus where editors are forbidden from copying the reference knowledge text; the metric's advantage over Levenshtein should come from its handling of substring reuse, so this test isolates whether that advantage is genuine.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the linear-time LZ factorization algorithm from suffix arrays used in the implementation."},{"cited_title":"Forcada, Miquel Espl \\`a -Gomis, and Lucia Specia","cited_arxiv_id":null,"evidence_quote":"Provides the IWSLT2019 post-editing dataset with keystrokes and edit times used for validation."}],"review_version":1}