Pith. sign in

REVIEW 4 major objections 5 minor 19 references

Measuring Information Distortion in Hierarchical Ultra long Novel Reconstruction:The Optimal Expansion Ratio

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 1% compression-expansion ratio best preserves meaning when rebuilding million-word novels, the paper argues.

desk verdict Useful million-word-scale study, but the R=0.01 optimality claim is contradicted by the paper's own Table 2 and needs a cost function and full-grid significance testing. read the letter →

arxiv 2505.12572 v2 pith:OW5TAHMU submitted 2025-05-18 cs.CL cs.AIcs.ITmath.IT

Hanwen Shen , Ting Ying This is my paper · ORCID
classification cs.CLcs.AIcs.ITmath.IT
keywords ultra-longnovelgenerationhierarchicaloutlinepipelinecompression-expansionratiorate-distortiontheorysemanticdistortionLLM-basedreconstructionChinesewebnovelsstructuredJSONoutlines
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks how much of a million-word novel's meaning survives being compressed into an outline and then expanded back by a large language model. It argues that the compression-expansion ratio—outline length divided by novel length—is the quantity that controls semantic and structural distortion, and that under its tested configuration the optimum is $R=0.01$: a 10,000-word outline for a 1,000,000-word novel. It further claims that the best pipeline is a mixed two-stage design with $\alpha_1=0.05$ (about 50,000 words) followed by $\alpha_2=0.20$ (about 10,000 words) and direct expansion from the global outline to chapter text, using structured JSON outlines. If correct, ultra-long novel generation can be steered by outlines as small as 1% of the original, and spending more than about 1% on the final outline buys little fidelity. The paper also finds that fidelity gains from enlarging the outline are steep from 0.1% to 1% but nearly flat from 1% to 2%.

What carries the argument

The load-bearing mechanism is a hierarchical two-stage compression-expansion pipeline with per-stage rates $\alpha_1$ and $\alpha_2$, combined with a composite distortion measure. The overall compression ratio is the product $R=\alpha_1\alpha_2$, and the objective is framed as a rate–distortion problem: minimize outline rate $R$ subject to reconstruction distortion below a bound $\epsilon$. Distortion is computed as a mixture of one minus cosine similarity, one minus BERTScore, an LLM-judged score across semantic, plot, character, background, and style similarity, and absolute differences in counts of characters, scenes, and props. The structured JSON outline—listing a plot-summary lead, characters, props, scenes, and foreshadowing—is what makes hierarchical compression reversible enough that direct expansion from the global outline outperforms staged expansion.

What would settle it

Score every chapter—not just eight per novel—of a set of completely reconstructed million-word novels and re-estimate distortion for the tested ratios $R=0.001, 0.005, 0.01, 0.015, 0.02$ and configurations. If the minimum distortion no longer falls at $R=0.01$ with $\alpha_1=0.05$, $\alpha_2=0.20$, and direct expansion, then the central optimal-ratio claim fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that under the tested setup—a specific large language model, Chinese web novels, and the paper's compression and expansion prompts—semantic and structural distortion in reconstruction is minimized at compression-expansion ratio $R=0.01$, meaning a global outline of about 10,000 words for a 1,000,000-word novel. The best-performing configuration is the mixed two-stage pipeline it calls K2-*: compress the novel first to a section-level outline at $\alpha_1=0.05$ (roughly 50,000 words), then to a global outline at $\alpha_2=0.20$ (roughly 10,000 words), and expand directly from the global outline to chapter text, skipping the intermediate outline. This configuration beats one-stage compression at the same ratio and beats full two-stage expansion on the paper's similarity and structural metrics. The paper also reports that distortion falls more steeply as the outline grows from 0.1% to 1% of the original than from 1% to 2%, so outline length has diminishing returns beyond the optimum.

Load-bearing premise

The conclusion assumes that distortion measured on eight sampled chapters per novel accurately represents whole-novel distortion, even though each million-word novel has roughly two hundred chapters.

Editorial extensions

If this is right

  • For the tested model and genre mix, a global outline of about 1% of the original length is enough; longer outlines add little fidelity, so practitioners can save tokens by stopping near $R=0.01$.
  • Compress hierarchically but expand directly: the K2-* shape beats both one-stage and full two-stage reconstruction at the same overall ratio.
  • Structured JSON outlines that track characters, props, scenes, and foreshadowing preserve more structure than plain narrative outlines of equal length.
  • Fidelity gains from outline length are strongly diminishing: the step from 0.1% to 1% helps more than doubling from 1% to 2%.
  • The claimed optimum is tied to the tested configuration; changing model, language, or prompts is expected to move the optimum.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's rate term $R=\alpha_1\alpha_2$ counts only the final global outline, not the intermediate section outline, so the literal end-to-end compression cost of the best configuration is about 6% of the original (50,000 words of section outline plus 10,000 words of global outline), not 1%; readers applying the ratio should budget accordingly.
  • The K2-* result suggests a design rule worth testing in other long-form generation settings: compression should be multi-level, but expansion should be single-step; applying that rule to textbooks, screenplays, or transcripts is a natural next experiment.
  • If the claimed $r=0.95$ correlation between sampled and full-book distortion holds on independent books, the eight-chapter sampling scheme would cut evaluation cost by roughly 96% and could become a reusable proxy for million-word-scale reconstruction studies.
  • Outlines larger than about 2% of the novel may approach the model's output-token ceiling and cause truncation rather than better fidelity; the paper does not test that regime, so the shape of the curve beyond $R=0.02$ is an open question.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper studies how the compression-expansion ratio R affects information distortion when reconstructing ultra-long Chinese web novels with Gemini 2.0 Flash. It compares a one-stage pipeline (novel to global outline to novel), a hierarchical two-stage pipeline (novel to section outline to global outline to novel), and a mixed two-stage pipeline (novel to section outline to global outline to novel, with the final expansion done directly from the global outline). The paper's central claims are that R=0.01 is the optimal compression-expansion ratio and that the mixed two-stage configuration K2-* (α1=0.05, α2=0.20) yields the best fidelity. The evidence consists of eight sampled chapters per novel across 40 novels, scored by GPT-4o on semantic, plot, character, background, and style similarity, plus structural difference metrics.

Significance. If the central claim were correct, the paper would provide a practically useful design rule for hierarchical long-form novel generation and an interesting empirical connection between rate-distortion ideas and LLM-based summarization and expansion. The paper has several strengths: it uses a substantial corpus of 40 million-word novels, evaluates with multiple complementary metrics, reports pairwise significance tests, and states its limitations candidly. However, the headline optimality claim is contradicted by the paper's own Table 2, and the sampling justification rests on an undescribed pilot study. As it stands, the paper does not establish the claimed optimum, so its main contribution is not supported.

major comments (4)
  1. [§1.2 and Table 2] The claim in Section 1.2 that 'R=0.01 is the optimal compression-expansion ratio under our configuration and experiment' is contradicted by Table 2: K2-* (R=0.010) has SemSim 0.613±0.216 and StyleSim 0.611±0.133, whereas K2-7 (R=0.020) has SemSim 0.710±0.166 and StyleSim 0.654±0.110, and K2-6, K2-8, and K2-9 (R=0.015-0.020) also exceed K2-* on these metrics. The table's boldface is explicitly restricted to 'tested configurations under R≤0.01', and the significance test in Section 4.1 compares only settings with R≤0.01; no pairwise test is reported against R>0.01 configurations. Without a stated cost constraint such as minimizing R subject to D≤ε or minimizing D+λR, the observed optimum lies at the tested boundary R=0.020, not R=0.010. The Limitations section explicitly declines to fix ε and excludes the intermediate outline cost, so the reported data do not support the headline optimality claim.
  2. [§4.1 Sampling Justification] The claim that sampled-chapter distortion approximates full-book distortion rests entirely on a pilot study over 40 books with r=0.95, but no details of the pilot, its protocol, or how full-book distortion was computed are provided. The Fisher z-transform sentence is asserted without the corresponding analysis. Since every reported distortion value is computed from only eight sampled chapters per novel, an unrepresentative pilot correlation would invalidate the measured optimal ratio for whole novels; this is load-bearing for the paper's central claim and needs to be documented or the sampling claim must be softened accordingly.
  3. [§2.1 and Limitations] The rate measure R=α1×α2 omits the token cost of the intermediate section outline, which is α1×L0. Consequently, comparisons between K=1 and K=2 configurations at the same nominal R do not compare equal total compression effort, and the 'optimal ratio' is not an optimal rate in the sense of the stated rate-distortion objective. The Limitations paragraph acknowledges that the intermediate cost is excluded but does not quantify the effect or provide a sensitivity analysis; for a paper whose title names an optimal expansion ratio, the excluded cost should enter the objective or be shown not to change the ranking.
  4. [§4.1, §6, and Appendix B] The paper's notion of 'optimal' is circular in practice: R is selected as the argmax of the measured similarity curve over a small grid, so the conclusion restates the best grid point rather than providing an independent prediction. Section 6 and Appendix B report only a weak correlation between R and similarity (r=0.129, p<0.001), which further undermines the idea that a sharp optimum at R=0.01 has been identified. The authors should either formulate a testable rate-distortion trade-off or explicitly reframe the contribution as a comparison of a small set of configurations.
minor comments (5)
  1. [Abstract and title] Please use 'ultra-long' consistently and fix the verb form in the abstract ('significantly reduce' should be 'significantly reduces').
  2. [Section 2.2 vs. Appendix J] The dataset description says 10 novels were chosen from each of four genres, while Appendix J describes PPS sampling with Neyman allocation; please reconcile these two descriptions or clarify how the 40 novels were actually selected.
  3. [Appendix J] The sentence 'We implement a two-stage sampling design :' is duplicated verbatim at the start of Appendix J and should be removed once.
  4. [Appendix F] The prompt template in Appendix F states a 200-300 word requirement for the plot summary introduction, but the accompanying JSON schema says '400-500 words'; please make these numbers consistent.
  5. [Section 4.1] There is a missing space in 'Bconfirms' in the first paragraph of Section 4.1.

Circularity Check

1 steps flagged · score 6.0 of 10

Central 'optimal R=0.01' claim is the argmax of the paper's own R≤0.01 table subset, not an independent derivation.

  1. fitted input called prediction [Section 1.2 (Contribution); Table 2 note; Section 4.1 significance testing]
    "R= 0.01 is the optimal compression-expansion ratio under our configuration and experiment. ... Bolded values indicate the best average performance for each metric among the tested configurations under R≤0.01, but do not imply statistical significance."

    The contribution statement is the paper's own table-selection rule renamed as a finding. The 'optimal' label is assigned to whichever configuration scored best within the R≤0.01 subset; concluding that R=0.01 (K2-*) is optimal is therefore the argmax of the same measured metric, not an independent prediction or a derived rate-distortion optimum. The Limitations explicitly state 'We do not fix a specific value of ε', so no cost function selects the R≤0.01 boundary. Removing that hidden filter, Table 2 shows K2-7 (R=0.020) has higher SemSim (0.710 vs 0.613) and StyleSim (0.654 vs 0.611) than K2-*, so the claimed optimum is an artifact of the restriction.

full rationale

The only load-bearing circularity is in the headline 'optimal ratio' claim. The paper does not derive R=0.01 from rate-distortion theory: the Limitations explicitly state ε is not fixed, and the intermediate outline token cost is excluded from the objective. The selection rule used to bold Table 2 is 'best among the tested configurations under R≤0.01'; the conclusion that K2-* / R=0.01 is optimal is therefore the same statement as the table's selection rule. This is a selected input reported as a prediction, not an independent result. No self-citation chain is used; the information-theoretic framing (R=α1×α2) is definitional and not circular by itself. The empirical distortion measurements are real and non-circular, so the circularity is partial: it concerns the labeling of the constrained argmax as 'optimal' without a cost function. Additionally, the paper's own Table 2 contradicts the global optimality claim (K2-7 at R=0.020 scores higher), which strengthens the conclusion that the R≤0.01 restriction, not an independent criterion, is what forces the claimed optimum.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four domain assumptions and a set of fitted hyperparameters. The most consequential are the word-count proxy for entropy and the sampling proxy for full-book distortion. No new theoretical entities are introduced.

free parameters (4)
  • R_opt = 0.01
    Selected as the best-performing grid point under R<=0.01; no cost function or independent test. Section 1.2 and Table 2.
  • alpha1 / alpha2 for K2-* = 0.05 / 0.20
    Chosen configuration that produces the claimed optimum; not derived from any theory. Section 5 and Table 3.
  • pilot correlation threshold = 0.90
    Used to validate the 8-chapter sampling proxy; the pilot study is not fully reported. Section 4.1.
  • LLM judge and generation settings = Gemini 2.0 Flash temp 0.3; GPT-4o judge
    Evaluation depends on these model choices; they are not varied or justified as representative. Section 3 and Appendix I.
assumptions (4)
  • domain assumption H(X) is defined as word count rather than Shannon entropy
    Stated in Limitations; this is a practical proxy, so the 'entropy' in the rate-distortion objective is not a true information measure. Section 2.1 and Limitations.
  • domain assumption Distortion D can be measured by cosine similarity, BERTScore, and GPT-4o scores
    The paper assumes LLM-based similarity and structural counts capture semantic distortion; acknowledged as potentially biased in Limitations and Section 2.4.
  • domain assumption The 8-chapter sample represents full-book distortion
    Justified only by an unreported pilot study (r=0.95 claimed). Section 4.1 and Appendix J.
  • domain assumption Gemini 2.0 Flash results generalize to other LLMs
    All experiments use one model; no cross-model validation is provided. Section 3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Measuring Information Distortion in Hierarchical Ultra long Novel Reconstruction:The Optimal Expansion Ratio." pith.science (2026). https://pith.science/paper/OW5TAHMU

@misc{pith2026250512572,
  author       = {Pith},
  title        = {Pith review of: Measuring Information Distortion in Hierarchical Ultra long Novel Reconstruction:The Optimal Expansion Ratio},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OW5TAHMU}},
  note         = {Machine review of arXiv:2505.12572}
}
read the original abstract

A two stage novel generation framework (outline -> section outline -> manuscript) is widely used in long novel generation,(e.g., \textsc{DOME}, \textsc{Plan\&Write}, \textsc{Long Writer}), but study of such framework in ultra long novel(>1M words) reconstruction is little. Building on recent text compression methods (\textsc{LLMZip}, \textsc{LLM2Vec}), we conduct an information-theoretic analysis to quantify semantic distortion under different compression-expansion ratios. We examine how outline length affects information preservation. Experiments on ultra-long novels show that the optimal compression-expansion ratio significantly reduces semantic distortion compared to other non-optimal compression-expansion ratio.

Figures

Figures reproduced from arXiv: 2505.12572 by the authors.

Figure 1
Figure 1. Pipeline for ultra-long novel generation using hierarchical outline approach. Our method maintains [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Pairwise significance test results between [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Correlation between the compression ratio [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Prompt templates used for LongWriter base [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Prompt templates used for our hierarchical [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Prompt templates used for hierarchical com [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Prompt templates used for hierarchical ex [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Prompt templates used for direct compression [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Prompt templates used for mixed hierarchi [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Prompt templates used for LLM-based eval [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 17 canonical work pages

  1. [1]

    Word count: approximately 5,000 words

    work page

  2. [2]

    plot summary introduction

    Maintain fluent writing style and vivid language. 【NOVEL OUTLINE】: {outline_text} 【Chapter {chap_num}】begins: Figure 4: Prompt templates used for LongWriter base- line reconstruction. English Translation is used for nov- els written in English. Chinese Translation is used for novels written in Chinese. D Appendix D：Prompts Templates： Hierarchical Prompts：...

    work page

  3. [3]

    Completely summarize the main storyline, especially main characters, core conflicts, key turning points and endings

    work page

  4. [4]

    Do not include chapter titles, numbers, or list symbols, narrate directly in natural paragraphs

    work page

  5. [5]

    Use only Simplified Chinese throughout

    work page

  6. [6]

    Word count≤1000 Chinese characters

    work page

  7. [7]

    While summarizing the full text, retain as many details, characters, and important plots as possible

    work page

  8. [8]

    【COMPLETE NOVEL TEXT】: {full_novel_text} Figure 8: Prompt templates used for direct compression method

    Tell me the title of the last chapter, this part does not count towards the 1000-word limit, as a test of your reading of the entire book Please strictly follow the above rules and output the completed book summary in one go. 【COMPLETE NOVEL TEXT】: {full_novel_text} Figure 8: Prompt templates used for direct compression method. This approach directly comp...

    work page

Show all 19 references
  1. [9]

    The following is

    Do not start with "The following is" or similar prompts, should directly enter the main text

  2. [10]

    Summarize the full text, note that this is to summarize a 1 million word novel from beginning to end

  3. [13]

    Vivid and coherent language

  4. [14]

    Word count must not be less than 5000 words

  5. [15]

    Focus on this chapter’s plot

  6. [16]

    Figure 9: Prompt templates used for mixed hierarchi- cal direct expansion method

    Output only the main text, no title. Figure 9: Prompt templates used for mixed hierarchi- cal direct expansion method. This approach directly expands from compressed outline to full chapter content (5000+ words) without intermediate structured analysis, providing a streamlined...

  7. [17]

    * If any category does not appear in the text, please return empty list ‘[]‘

    Extract from Text A and Text B respectively: - List of props appearing (e.g., sword, jade, token, etc.) - List of character names appearing - List of scene/environment names appearing * Please extract as precisely as possible, remove generic words (e.g., ’person’, ’place’), ke...

  8. [18]

    Count the number of each type of element (after deduplication) and output the element list and count for each category

  9. [19]

    **Please strictly output the following JSON format, do not include markdown “‘json

    Then compare the two text contents and score on the following 5 dimensions from 0-1 (1 means very similar, 0 means completely different): - semantic_similarity Overall semantics/theme - plot_similarity Plot and event development - character_similarity Character names, quantity...

    2013

  10. [2018]

    Deepzip: Lossless data compression using recurrent neural networks.Preprint, arXiv:1811.08162. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Ka- dian, A. Al-Dahle, A. Letman, A. Mathur, A. Schel- ten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra,...

    2024 arXiv

  11. [2024]

    Li Yuan, Shuai Gao, and Peng Pan

    Semantic similarity matching for patent docu- ments using ensemble bert-related model and novel text processing method.Preprint, arXiv:2401.06782. Li Yuan, Shuai Gao, and Peng Pan. 2023. Ctsarf: A chi- nese text similarity analysis model based on residual fusion.Neurocomputing...

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.