Pith. sign in

REVIEW 5 major objections 5 minor 21 references

Assessing Human Editing Effort on LLM-Generated Texts via Compression-Based Edit Distance

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Counting LZ77 phrase differences between an original LLM answer and its edited version correlates with human editing time at up to r = 0.81, exceeding standard automatic metrics.

desk verdict Useful dataset, overclaimed novelty, and a statistics problem: the headline result is likely inflated by a length confound and the metric is prior art, but the paper deserves a serious referee for its empirical contribution. read the letter →

arxiv 2412.17321 v1 pith:UHOJOOR4 submitted 2024-12-23 cs.CL cs.AI

Nicolas Devatine , Louis Abraham This is my paper · ORCID
classification cs.CLcs.AI
keywords compressiondistanceLZ77editpost-editingeffortLLMevaluationhuman-in-the-loopsegmentrearrangementtext
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that the amount of human effort spent editing an LLM-generated text can be measured as a difference in LZ77 compression phrase counts between the original and the edited text. This is the compression-based edit distance $d(S\to T)=\mathrm{LZ}(S\mid T)-\mathrm{LZ}(S)$, and the authors argue it captures the block moves, deletions, and copy-paste reuse that dominate real post-editing but are missed by BLEU, ROUGE, TER, and BERTScore. They support the claim with a new dataset of 200 French accounting answers generated by an LLM, each edited by four expert annotators with recorded edit times, plus synthetic edits under normal, structure-preserving, and fast instructions. On this data the metric's Pearson correlation with edit time reaches 0.81 and a 5-nearest-neighbour regressor using only the metric predicts edit time with $R^2=0.63$, roughly double the best baseline. If the result is right, it gives a parameter-free, linear-time tool for estimating how much human work a model's output will require.

What carries the argument

The load-bearing object is the LZ77 factorization of a string, the greedy parsing into the longest substrings that have already occurred earlier in the text. The paper's metric is the difference $d(S\to T)=\mathrm{LZ}(S\mid T)-\mathrm{LZ}(S)$ in phrase counts, which measures how much new information the edited text contributes once the original has already been parsed. The theoretical justification is a constant-factor approximation result connecting this quantity to the minimum number of character-level and substring-level edit operations (moves, copies, deletions), and the practical engine is a linear-time algorithm that computes the factorization from suffix arrays. This machinery lets the distance treat a copied sentence from a reference text as cheap, while still charging for deletions, reorderings, and fresh content—behaviour that aligns with how human post-editors actually work.

What would settle it

Recompute the correlation between compression distance and edit time after partialling out source-text length (or fitting a mixed-effects model with a random intercept for each of the 200 source texts). If the partial correlation falls to near zero, the claim that the metric tracks editing effort collapses. A second check is to run the same correlation on a held-out corpus where editors are forbidden from copying the reference knowledge text; the metric's advantage over Levenshtein should come from its handling of substring reuse, so this test isolates whether that advantage is genuine.

Watch

Extended reading notes

Core claim

The paper establishes that post-editing effort is well approximated by a compression distance defined through the LZ77 parsing: $d(S\to T)=\mathrm{LZ}(S\mid T)-\mathrm{LZ}(S)$, where $\mathrm{LZ}$ counts phrases in the greedy factorization. This quantity upper-bounds, within a constant factor of 4, the minimum number of character edits, substring relocations, duplications, and deletions required to transform the original into the edited text, and it can be computed in linear time using a suffix-array-based LZ factorization. Empirically, on the paper's human-edited accounting answers the distance correlates with measured edit time at $r=0.71$ without and $r=0.81$ with the expert knowledge text concatenated to the original, outperforming Levenshtein, CharacTER, TER, BLEU, ROUGE-L, METEOR, and BERTScore. On the IWSLT2019 post-editing benchmark the metric tracks keystrokes ($r\approx 0.82$ to $0.89$ across five annotators) and edit time nearly as well as Levenshtein distance, and synthetic experiments show that edits produced under 'fast' or structure-preserving instructions have systematically lower compression distances. The authors conclude that the metric reflects both the mechanical and the cognitive load of post-editing, and that language models encode the notion of editing speed in a way that matches this compression-based measure.

Load-bearing premise

The load-bearing assumption is that the strong correlation between compression distance and edit time is not an artifact of both quantities growing with the length of the source text—the same 200 texts were edited by all four annotators, yet the observations are treated as independent in the reported Pearson correlations.

Editorial extensions

If this is right

  • Post-editing effort on LLM output can be scored in linear time without any trained model or similarity embeddings, making it usable on very large document collections.
  • Because the metric charges little for reuse of a reference knowledge base, it agrees with the observation that editors often copy sentences from supplied sources; traditional metrics penalize that reuse and therefore mis-rank editing effort.
  • The synthetic experiments show that edits requested under 'similar' or 'fast' instructions yield lower compression distances, so the metric could be used to diagnose how deep an editing pass actually was.
  • On the IWSLT2019 benchmark, the compression distance is competitive with Levenshtein distance for keystroke and time prediction and clearly better than TER, BLEU, METEOR, and BERTScore, which suggests the advantage is not specific to the new dataset.
  • Combining the metric with others in a KNN regressor did not improve prediction beyond the metric alone, suggesting the compression distance already captures most of the editing-effort signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the length-confounding worry is resolved, the metric could be adopted outside LLM evaluation—for example, estimating revision effort in document management or translation workflows, where block operations and copy-paste dominate.
  • The sensitivity to substring reuse suggests a practical testable use: flagging edits that are pure copy-paste from a reference source as low-effort, which could be useful for quality control of outsourced editing.
  • A natural extension would be to check whether the 4-approximation's correlation with edit time persists when edits are performed by non-experts or in languages without the strong reference-knowledge setup used here, since the accountancy dataset has a very specific edit regime.
  • The metric's link to editing speed could give a principled way to calibrate LLM prompts: the 'fast edit' scenario systematically reducing compression distance implies the model internalizes effort not just as token count but in a way consistent with substring-level operations.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces a compression-based edit distance, d(S→T) = LZ(S|T) − LZ(S) (Section 3), grounded in LZ77 factorization, intended to measure the human effort of post-editing LLM-generated texts. The authors contribute a new dataset of 200 accounting questions with human edits by four expert annotators and synthetic LLM edits under three instructed scenarios, and they evaluate their metric against Levenshtein, TER, CharacTER, BLEU, ROUGE-L, METEOR, and BERTScore on this dataset and on IWSLT2019. The headline result is a Pearson correlation of 0.81 with human edit time when the expert knowledge text is concatenated to the source, compared with 0.59 for Levenshtein under the same condition. The paper also reports KNN regression results and synthetic-edit analyses intended to show that the metric tracks editing intensity and that LLMs' notion of editing speed aligns with the distance.

Significance. If the empirical claims are supported by sound statistics, this is a valuable contribution: the metric is parameter-free, has a clear information-theoretic interpretation, and runs in linear time via the Crochemore et al. algorithm, which is a practical advantage over BERTScore and other expensive metrics. The authors provide code and data, and the metric definition is transparent and reproducible. However, the central claim of superiority over existing metrics is currently under-supported because the key comparison (Table 1) uses an analysis that does not account for clustering or text-length confounding, and the reported advantage depends on a particular concatenation protocol that can differentially favor compression-based measures. The IWSLT2019 results actually show compression distance is roughly on par with Levenshtein, so the only place where compression distance clearly outperforms baselines is the analysis most vulnerable to confounding.

major comments (5)
  1. [Section 4.2, Table 1] The Pearson correlations in Table 1 are computed on the pooled 800 observations (4 annotators × 200 source texts) as if they were independent, but the observations are clustered by source text: each of the 200 generated answers is edited by all four annotators, so observations sharing a source text are correlated. Both edit time and the compression distance d(S→T) scale with text length, and source length varies substantially (mean 1008, SD 260, Figure 2). A pooled Pearson correlation can therefore be inflated by the shared length factor even if the metric has no per-edit-effort signal. The paper does not report partial correlations controlling for source length, within-source centering, mixed-effects models, or cluster-robust significance tests. The claim that compression distance outperforms Levenshtein (0.81 vs. 0.59) is not supported by the presented statistics. Please re-analyze with hierarchical or cluster-robust methods, or provide per-source averaged correlations, and report confidence intervals.
  2. [Section 4.2, W/Knowledge condition] The comparison labeled "W/ Knowledge" concatenates the expert knowledge text K to the source S only, leaving the target T as the edited answer. This asymmetric construction is likely to favor compression distance disproportionately, because LZ-based measures are specifically sensitive to long repeated substrings and can detect copied knowledge fragments in T, while Levenshtein and other character/word-level metrics do not exploit such block-level repetitions to the same degree. The paper interprets the improved correlation as evidence that human editors reuse knowledge text, but the protocol itself may be the source of the improvement. To support the claim, the authors should either (i) justify the concatenation protocol on independent grounds and compare it against concatenating K to both S and T, or (ii) report the W/o Knowledge result as the primary comparison and treat the W/Knowledge result as a secondary, sensitivity analysis. As it stands, the key advantage in Table 1 is concentrated in a condition that is confounded with the metric's design.
  3. [Section 4.2, Table 2] On the IWSLT2019 dataset, where correlations are reported per annotator (avoiding the clustering issue), compression distance does not outperform Levenshtein: for edit-time correlations, the ranges are 0.55–0.73 for compression distance and 0.55–0.73 for Levenshtein, with Levenshtein winning on A1 and A4 and compression distance winning on A0, A2, and A3. The paper attributes the parity to the absence of block operations in the PET tool; however, this substantially weakens the general claim that compression distance 'outperforms' existing metrics. The authors should either temper the claim or provide a more direct test of block-operation sensitivity, e.g., by introducing controlled block edits into a subset of the IWSLT2019 data and showing a differential gain.
  4. [Section 4.2, Synthetic Edits] The synthetic-edits analysis (Figure 3) demonstrates that the compression distance is smaller in the 'fast' and 'similar' scenarios than in the 'normal' scenario, which is consistent with the authors' expectations. However, no baseline metric is evaluated on the same synthetic data. It is unclear whether this ordering is specific to compression distance or would also be observed with Levenshtein, TER, or even simple length difference, since the scenarios are defined by instructions that likely reduce the amount of textual change. Without a baseline comparison, the claim that 'LLMs exhibit an implicit understanding of editing speed that aligns well with our metric' (Abstract) is not established. Please add baseline metric values and, if possible, a statistical test (e.g., paired comparison across questions) for the scenario-ordering effect.
  5. [Section 4.2, KNN regression] Tables 3 and 4 report KNN regression R² values on random 80/20 splits of the data. For the human-edited dataset (Table 4), the 800 observations again are not independent because of the 4-annotator-per-source design; a random split can place the same source text in both training and test sets, inflating the KNN R² for all metrics, especially those that are sensitive to text length. This may explain the very high R² (0.6316) for compression distance. Please evaluate using a grouped split (e.g., all annotations of a source in the same fold) or report per-source cross-validation, and also report R² for a length-only baseline to check for confounding.
minor comments (5)
  1. [Section 3] The definition of d(S→T) uses the concatenation 'S | T' but the delimiter is not specified; the choice of delimiter can affect LZ77 factorization and should be stated explicitly, along with whether the delimiter is a special character or a whitespace token.
  2. [Section 4.2] The text says 'All reported correlation coefficients are statistically significant (p <0.05)', but no test procedure, degrees of freedom, or correction for multiple comparisons is described; please specify the significance test and whether cluster-robust p-values were used.
  3. [Section 5] In the paragraph beginning 'From Table 1', there is a typo: 'suggesting thatn-gram overlap' should be 'suggesting that n-gram overlap'.
  4. [Section 3] The sentence 'We apply the algorithm proposed by Crochemore et al. (2008) for computing the Lempel-Ziv factorization in linear time from suffix arrays. as implemented in pydivsufsort2' begins with a capital 'We' but the second sentence starts with 'as', which is a sentence fragment and missing a period; please correct the punctuation.
  5. [Section 4.1.1] Figure 1 is referenced in the text before it appears in the provided PDF; please ensure the figure placement is consistent in the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the metric is defined independently of the edit-time data, and the reported correlations are empirical validations rather than derived predictions.

full rationale

The paper's central object, d(S→T) = LZ(S|T) − LZ(S), is a parameter-free compression-based distance defined solely from the source and target strings; it is not fitted to edit times, keystrokes, or any other measured human-effort variable. The theoretical grounding is an approximation bound attributed to Ergün et al. (2003), an external result, and the linear-time factorization is attributed to Crochemore et al. (2008); the only self-citation is to Abraham (2023) for a suffix-array implementation, which is a computational dependency rather than a load-bearing justification of the metric's validity. The Pearson correlations in Tables 1–2 and the KNN R² values in Tables 3–4 are empirical evaluations of the metric against independent human-edit data, not derivations of the metric from that data. The W/Knowledge condition is an evaluation design choice that the authors motivate by copy-paste behavior; while the choice may favor a compression-based metric, it does not make the correlation a tautology or reduce the result to the metric's definition. The synthetic 'fast edit' experiment is likewise an empirical consistency check: the LLM is instructed to edit quickly, and the resulting compression distance is observed to be smaller. This test does not prove the metric by construction, because the prompt does not mention compression distance; however, it is a weak inference on its own, since faster instructions could also reduce other edit-count measures. The pooled correlation analysis ignoring source-text length and annotator clustering is a legitimate statistical concern about significance and effect size, but that is a correctness and robustness issue, not circularity. No step in the paper's derivation chain equates the claimed prediction with its inputs, and no load-bearing claim rests on the authors' own prior work.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The metric has no fitted parameters; K=5 in the KNN regressor is a hyperparameter for evaluation, not part of the metric. The paper introduces no new entities, only a metric and a dataset.

assumptions (3)
  • domain assumption Ergün et al. (2003) theorem that the LZ-based compression distance approximates the segment rearrangement distance within a constant factor.
    Invoked in Section 3 to justify the metric as an edit-distance proxy; the paper does not reprove the theorem, and the central claim relies on it being valid.
  • standard math Crochemore et al. (2008) algorithm computes the LZ77 factorization in linear time.
    Used in Section 3 to claim linear-time computation; assumed correct without independent verification in this paper.
  • domain assumption Human editing time measured by annotators is a valid ground truth for editing effort.
    The evaluation compares metric values against measured times; if annotator times are noisy or biased, the correlation findings are weakened.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assessing Human Editing Effort on LLM-Generated Texts via Compression-Based Edit Distance." pith.science (2026). https://pith.science/paper/UHOJOOR4

@misc{pith2026241217321,
  author       = {Pith},
  title        = {Pith review of: Assessing Human Editing Effort on LLM-Generated Texts via Compression-Based Edit Distance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UHOJOOR4}},
  note         = {Machine review of arXiv:2412.17321}
}
read the original abstract

Assessing the extent of human edits on texts generated by Large Language Models (LLMs) is crucial to understanding the human-AI interactions and improving the quality of automated text generation systems. Existing edit distance metrics, such as Levenshtein, BLEU, ROUGE, and TER, often fail to accurately measure the effort required for post-editing, especially when edits involve substantial modifications, such as block operations. In this paper, we introduce a novel compression-based edit distance metric grounded in the Lempel-Ziv-77 algorithm, designed to quantify the amount of post-editing applied to LLM-generated texts. Our method leverages the properties of text compression to measure the informational difference between the original and edited texts. Through experiments on real-world human edits datasets, we demonstrate that our proposed metric is highly correlated with actual edit time and effort. We also show that LLMs exhibit an implicit understanding of editing speed, that aligns well with our metric. Furthermore, we compare our metric with existing ones, highlighting its advantages in capturing complex edits with linear computational efficiency. Our code and data are available at: https://github.com/NDV-tiime/CompressionDistance

Figures

Figures reproduced from arXiv: 2412.17321 by the authors.

Figure 1
Figure 1. Overview of our dataset construction pro [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Length distribution of initial LLM answers and distribution of the compression-based edit distances for [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of compression distances on our [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Scatter plots with linear regression fits and confidence intervals for various metrics against measured edit [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 8 canonical work pages

  1. [1]

    Louis Abraham. 2023. https://doi.org/10.5281/zenodo.7932458 pydivsufsort . https://github.com/louisabraham/pydivsufsort

    work page doi:10.5281/zenodo.7932458 2023

  2. [2]

    Anthropic. 2024. Introducing the next generation of claude. https://www.anthropic.com/news/claude-3-family

    work page 2024

  3. [3]

    Wilker Aziz, Sheila Castilho, and Lucia Specia. 2012. http://www.lrec-conf.org/proceedings/lrec2012/pdf/985_Paper.pdf PET : a tool for post-editing and assessing machine translation . In Proceedings of the Eighth International Conference on Language Resources and Evaluation ( LREC '12) , pages 3982--3987, Istanbul, Turkey. European Language Resources Asso...

    2012

  4. [4]

    Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909 METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan. Association for Comput...

    2005

  5. [5]

    Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. https://aclanthology.org/E06-1032 Re-evaluating the role of B leu in machine translation research . In 11th Conference of the E uropean Chapter of the Association for Computational Linguistics , pages 249--256, Trento, Italy. Association for Computational Linguistics

    work page 2006

  6. [6]

    Rudi Cilibrasi and Paul Vitanyi. 2004. https://arxiv.org/abs/cs/0312044 Clustering by compression . Preprint, arXiv:cs/0312044

    work page Pith review arXiv 2004

  7. [7]

    Maxime Crochemore, Lucian Ilie, and W. F. Smyth. 2008. https://doi.org/10.1109/DCC.2008.36 A simple algorithm for computing the lempel ziv factorization . In Proceedings of the Data Compression Conference, DCC '08, page 482–488, USA. IEEE Computer Society

    work page doi:10.1109/dcc.2008.36 2008

  8. [8]

    Funda Ergün, Senthilmurugan Muthukrishnan, and Cenk Sahinalp. 2003. https://doi.org/10.1007/978-3-540-24597-1_16 Comparing sequences with segment rearrangements . pages 183--194

    work page doi:10.1007/978-3-540-24597-1_16 2003

Show all 21 references
  1. [9]

    Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2022. https://arxiv.org/abs/2202.06935 Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text . Preprint, arXiv:2202.06935

    2022 arXiv

  2. [10]

    Zhiying Jiang, Matthew Yang, Mikhail Tsirlin, Raphael Tang, Yiqin Dai, and Jimmy Lin. 2023. https://doi.org/10.18653/v1/2023.findings-acl.426 `` low-resource '' text classification: A parameter-free classification method with compressors . In Findings of the Association for Co...

    2023 doi

  3. [11]

    Levenshtein

    Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8):707--710

    1966

  4. [12]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  5. [13]

    George A. Miller. 1994. https://aclanthology.org/H94-1111 W ord N et: A lexical database for E nglish . In H uman L anguage T echnology: Proceedings of a Workshop held at P lainsboro, N ew J ersey, M arch 8-11, 1994

    1994

  6. [14]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  7. [15]

    Forcada, Miquel Espl \`a -Gomis, and Lucia Specia

    Scarton Scarton, Mikel L. Forcada, Miquel Espl \`a -Gomis, and Lucia Specia. 2019. https://aclanthology.org/2019.iwslt-1.23 Estimating post-editing effort: a study on human judgements, task-based and reference-based metrics of MT quality . In Proceedings of the 16th Internatio...

    2019

  8. [16]

    Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. https://aclanthology.org/2006.amta-papers.25 A study of translation edit rate with targeted human annotation . In Proceedings of the 7th Conference of the Association for Machine Translation ...

    2006

  9. [17]

    Lucia Specia and Atefeh Farzindar. 2010. https://aclanthology.org/2010.jec-1.5 Estimating machine translation post-editing effort with HTER . In Proceedings of the Second Joint EM+/CNGL Workshop: Bringing MT to the User: Research on Integrating MT in the Translation Industry, ...

    2010

  10. [18]

    Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016. https://doi.org/10.18653/v1/W16-2342 C harac T er: Translation edit rate on character level . In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 505--510,...

    2016 doi

  11. [19]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert . Preprint, arXiv:1904.09675

    2020 arXiv

  12. [20]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  13. [21]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.