{"id":"1f0dff46-1369-4b01-aa9c-e6e43da98cd2","arxiv_id":"2505.24319","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"HiCaM combines an LLM-built hierarchical summary tree with a causal entity graph to guide long-document editing, reporting 56.8-72.5% win rates and up to 59.5% net win rates over direct LLM baselines as judged by GPT-4o.","lead":"This paper introduces HiCaM, a prompting framework that first builds a hierarchical summary tree and a causal entity graph, then uses them to guide an LLM's long-document editing, reporting win rates of 57% to 79% over direct LLM baselines. It matters because it offers a training-free recipe for more coherent editing, and because the evaluation shows how much LLM-judge evidence still needs human verification.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Undisclosed scaffolding model confounds Table 2: if GPT-4o builds the tree/graph for weaker baselines, win rates reflect a stronger planner, not HiCaM.","rationale":"The central empirical claim is that augmenting four LLMs with HiCaM improves long-form text modification. The comparison in Table 2 is only meaningful if the augmented pipeline for a given baseline uses that same model's own capabilities. The paper never states this. The repeated phrase 'we use a large language model (LLM)' for entity extraction (2.1), causal graph construction (2.2.2), and the final modification (2.3) leaves open the possibility that a single strong model (e.g., GPT-4o) performs all the scaffolding for all four baselines. In that case, the results would show that a weaker generator benefits from a stronger planner's analysis, not that HiCaM is a model-agnostic plug-in. This confound is more load-bearing than the LLM-judge issue because it threatens the internal validity of the comparison rather than the measurement accuracy: even a perfect judge would not rescue the claim if the augmented system has access to a stronger model. The paper does have strengths: the causal-graph ablation (Table 4) shows substantial drops on logically structured datasets, the length-bias control (Table 12) is an appropriate check, and the dataset spans multiple domains. These do not, however, resolve the model-identity question. The reader's concern about GPT-4o-as-judge without human agreement is valid, but it is secondary: the effect sizes are large, and the direction of any judge bias is unclear, whereas a stronger scaffolding model would produce exactly the observed pattern of larger gains on weaker models (GPT-4o-mini shows the largest net win rate, 39.40%), consistent with the 'weaker generator + stronger planner' explanation. In addition, the Introduction's reported win-rate range (56.81–72.47%) contradicts Table 2's 79.50% for GPT-4o-mini on MultiFieldQA-zh, an internal inconsistency that reinforces the need for the missing disclosures. The conditional verdict stands: the paper should be accepted only if the authors disclose the scaffolding model per condition, and ideally rerun with same-model scaffolding to verify the win rates.","tokens_in":18697,"tokens_out":9476,"duration_ms":120403,"concrete_test":"Obtain or reverse-engineer the exact prompt templates and any released code for the pipeline; identify the model identifier used in the scaffolding calls for each condition in Table 2. If the scaffolding model is not identical to the generator for each row, rerun the experiments with the scaffolding model set equal to the generator (e.g., GPT-4o-mini performs entity extraction, tree building, and graph construction for the GPT-4o-mini condition). If the win rates drop materially, the headline improvement is attributable to the stronger scaffolding model, not to HiCaM.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper never identifies which LLM performs the scaffolding steps (entity extraction, summary-tree construction, causal-graph construction). Sections 2.1 and 2.2.2 state only 'we use a large language model (LLM)'; Section 4.1 says only 'During the pipeline, a temperature value of 0.7 is applied to the model.' If the scaffolding is executed by a fixed strong model (e.g., GPT-4o) while the generator is the baseline under test (e.g., GPT-4o-mini, DeepSeek-V3, QWQ-32B), then the augmented system is not 'model M + HiCaM'; it is 'model M + GPT-4o's analysis.' The Table 2 win rates then conflate the contribution of the framework with the contribution of a stronger planner, so the central claim—that HiCaM consistently improves each LLM's own long-form editing—is not established. The ambiguity is compounded by an internal reporting error: the Introduction states win rates range from 56.81% to 72.47%, but Table 2 reports GPT-4o-mini at 79.50% on MultiFieldQA-zh. These problems together mean the empirical core of the paper cannot be independently checked without the missing disclosure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HiCaM, a training-free framework for long-form text modification. Given an original text and a modification suggestion, HiCaM first extracts key entities, optionally chunks the text, and then builds two entity-centric structures: a hierarchical summary tree whose nodes mark text spans with summaries and local modification instructions, and a causal graph of directional entity relations. The final modified text is generated by prompting an LLM with tree-structured suggestions derived from these structures. The authors also construct an evaluation dataset by taking documents from seven existing benchmarks and generating modification suggestions with an LLM. They report pairwise GPT-4o-judged comparisons of HiCaM-augmented versus direct-prompt baselines for four LLMs, with win rates mostly in the 56-72% range (with one 79.50% cell), and ablation studies varying chunk size, causal graph inclusion, and entity filtering.","tokens_in":18912,"tokens_out":4930,"duration_ms":58713,"significance":"If the empirical claims hold, HiCaM would be a useful plug-and-play contribution: it requires no training, is model-agnostic, and addresses two clearly motivated failure modes (over-editing and missed implicit updates). The dataset, though LLM-generated, is a plausible first resource for a task that currently lacks benchmarks. The causal-graph ablation (Tables 4-5) shows substantial drops on logically structured datasets, which is a nice falsifiable component. However, the current evidence is almost entirely internal: the evaluation set, the modification suggestions, and the quality judgments are all produced by LLMs, with no human validation and no disclosed identity of the scaffolding model. The claimed consistency across models and domains is therefore not yet established at the level claimed.","major_comments":[{"comment":"The manuscript never identifies which LLM performs entity extraction, tree construction, and causal-graph construction; it only says 'we use a large language model (LLM)' and that a temperature of 0.7 is applied 'to the model.' If a fixed strong model (e.g., GPT-4o) builds the structures for all four baselines, then Table 2 does not compare 'model M + HiCaM' against model M; it compares 'model M + scaffolding-by-GPT-4o + final generation by M' against direct M. The win rates would then partially reflect the scaffolding model's planning ability rather than the framework itself. Please specify the scaffolding model for each baseline, or run the scaffolding with each baseline model, and report both variants.","section":"§2.1, §2.2.2, §4.1"},{"comment":"The evaluation is fully internal. The modification suggestions in the dataset are LLM-generated and never human-validated (Section 3), and the pairwise quality judgments are made by GPT-4o with no human agreement statistics. GPT-4o also serves as a baseline and, as far as can be told, as the scaffolding model, so self-enhancement and style preferences cannot be ruled out by the statement in Appendix E that 'we observe no evidence of bias toward its own outputs' without reporting numbers. Please add a human-annotated sample (e.g., 100-200 pairs across datasets) with inter-annotator agreement, and report judge agreement with GPT-4o.","section":"§3, §4.1, Appendix E"},{"comment":"The reported win-rate range is internally inconsistent. The Introduction states 'win rates ranging from 56.81% to 72.47%, with net win rates between 13.87% and 59.50%,' but Table 2 reports GPT-4o-mini on MultiFieldQA-zh at 79.50% win rate, and the Abstract claims 'up to a 79.50% win rate.' Please reconcile these statements; as written, the headline ranges exclude a cell in the paper's own main table, which matters because the central claim is 'consistent performance improvements.'","section":"Introduction, Abstract, Table 2"},{"comment":"The evaluation units are undefined and the counts do not match the dataset sizes. For example, Table 1 reports QASPER with 200 documents, but the Deepseek-V3 row in Table 2 sums to 382 pairwise judgments; MultiFieldQA-zh has 200 documents but the GPT-4o-mini row sums to 400 judgments. The paper does not state how many modification suggestions per document were generated or how multiple judgments on the same document were aggregated. Please define the evaluation unit (document, suggestion, or judgment), state whether judgments on the same document are independent, and report confidence intervals or significance tests for the win rates.","section":"Table 2 vs Table 1"}],"minor_comments":[{"comment":"The stopping condition uses a threshold τ but never defines its value; please state the default and whether it was tuned.","section":"Algorithm 1, line 22"},{"comment":"Table 13 lists LVEval and LongBench as sources, but Table 1 and Section 3 list a different set of benchmark datasets; clarify the mapping (for instance, is MultiFieldQA drawn from LongBench?).","section":"Table 13 vs Table 1"},{"comment":"The length-ratio analysis is an indirect proxy for undesired modifications; consider reporting direct evidence such as overlap between the spans that were modified and the spans that should have been preserved.","section":"Figure 5"},{"comment":"The sentence 'During the pipeline, a temperature value of 0.7 is applied to the model' does not say whether this applies to scaffolding, final generation, or both; please clarify.","section":"§4.1"},{"comment":"The cross-reference 'Table B.2' should point to Table 12; the table is numbered inconsistently.","section":"Appendix B.3"},{"comment":"The term 'net win rate' is used without definition; please define it explicitly (presumably (Win - Lose) / total judgments).","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's evaluation design is the main risk: the undisclosed scaffolding model, the fully LLM-generated dataset, and the LLM-only judge make the central empirical claim hard to verify. I would push the authors to disclose scaffolding details and add a human-validated evaluation subset before considering acceptance. The topic fits cs.CL well, and the framework itself is plausible, but the current evidence is not yet at the level claimed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is a reasonable engineering contribution with a serious reporting gap. The central claim—that HiCaM consistently improves four LLMs on long-form editing—cannot yet be accepted because the paper never discloses which LLM performs the scaffolding (entity extraction, summary-tree construction, causal-graph extraction) in the pipeline. If that scaffolding is done by a stronger model than the generator, the win rates are not attributable to HiCaM. This is the load-bearing ambiguity.\n\nWhat's actually new: the specific combination of a hierarchical entity-oriented summary tree with an entity-level causal graph for propagating edits across implicit dependencies. Each component is familiar from GraphRAG-style summarization and graph extraction, but the integration for text modification is new, and the eight-dataset benchmark is a useful resource. The method is training-free, the causal-graph ablation shows meaningful drops on logically structured datasets, and the length-bias control in Appendix B.3 is a decent internal check.\n\nThe soft spots are real. First and most serious: the undisclosed scaffolding model. Second, the evaluation is entirely internal—self-built dataset, LLM-generated modification suggestions with no human validation, GPT-4o as judge, no human agreement statistics, no confidence intervals, no significance tests. Third, there is an internal inconsistency: the introduction says win rates range from 56.81% to 72.47%, but Table 2 reports GPT-4o-mini at 79.50% on MultiFieldQA-zh, matching the abstract. Fourth, the hierarchical tree is built with depth limit 1, which is shallow for a method whose name promises hierarchy.\n\nWho this is for: people working on controllable generation, long-document editing, or training-free wrappers. It deserves serious peer review—the idea is plausible and the benchmark could be genuinely useful—but only with the expectation of major revision.\n\nRecommendation: send it to reviewers, and direct them to the evaluation protocol and the scaffolding disclosure. If those are not fixed, reject.","headline":"Useful training-free editing idea, but the win rates are uninterpretable until the scaffolding model is disclosed and the evaluation is anchored by human judgment.","tokens_in":19462,"tokens_out":3465,"would_cite":false,"duration_ms":41914,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structuring a document as an entity summary tree plus a causal graph before editing makes LLMs revise long texts more faithfully than direct prompting, with win rates up to 79.50%.","keywords":["long-form text modification","hierarchical summary tree","causal graph","entity-centric structure","LLM-as-judge evaluation","multi-domain benchmark","training-free framework","controlled text generation"],"falsifier":"Run a human preference study on a few hundred of the benchmark's comparison pairs: with readers blind to which output came from which method, compare their choices with GPT-4o's verdicts—if agreement is near chance, the reported win rates measure the judge's tastes rather than editing quality. A complementary check targets the mechanism directly: because HiCaM promises to leave irrelevant spans untouched, measure the fraction of unedited regions kept byte-for-byte in HiCaM outputs versus direct-prompt outputs.","tokens_in":18457,"feed_emoji":"✍️","tokens_out":13893,"duration_ms":134453,"temperature":0.7,"pith_summary":"This paper claims that large language models fail at long-form text modification in two predictable ways—rewriting or summarizing content that should stay untouched, and missing updates to passages only implicitly connected to the requested change—and that both failures can be fixed without any retraining. The fix, called HiCaM, pre-structures a document along two axes before any editing happens: a hierarchical summary tree that maps every text region to the entities it concerns, and a causal graph of directed influence relations among the key entities. Edits are then planned node by node in the tree, while the causal graph forces each change to propagate through every passage it logically touches, so untouched regions stay verbatim. On a new multi-domain benchmark assembled from seven existing long-text corpora, the paper reports that four LLMs (GPT-4o, GPT-4o-mini, Deepseek-V3, QWQ-32B) augmented with HiCaM win pairwise comparisons against direct prompting at rates up to 79.50%. The claim matters because over-editing and under-editing are exactly the failure modes that make LLM editing of stories, protocols, and manuscripts impractical today.","feed_headline":"Entity maps lift long-text editing win rates to 79 percent","feed_subtitle":"Planning edits from a summary tree plus causal graph beats prompt-only editing across four LLMs and eight datasets.","key_machinery":"The load-bearing objects are two analytical structures built before generation. The hierarchical entity-oriented summary tree is a recursive decomposition of the text in which each node covers a span (marked by opening and closing phrases), carries a summary focused on the key entities, and either proposes a modification for its span or is omitted entirely; its job is to confine edits to relevant regions so that irrelevant content survives verbatim. The causal graph is a set of directed edges among entities, each edge labeled with the kind of influence ('causes,' 'depends on,' 'affects'), built chunk-by-chunk and merged into a global graph; its job is to convert implicitly relevant passages into explicitly tracked dependencies so that a change to one entity propagates through all downstream relations. The third component is the tree-structured modification suggestion itself, the joint artifact that turns the two structures into a concrete editing plan for the final pass.","core_discovery":"The central claim is that the two observed failure modes of long-form text modification are structural, not parametric: the model does not know which regions are safe and which passages depend on one another. HiCaM supplies both pieces of knowledge explicitly. It first extracts the entities named in the modification suggestion, each with an importance score and a description of the intended change, in a chain-of-thought-inspired step. It then builds, in parallel, a hierarchical entity-oriented summary tree, which recursively decomposes the text into entity-anchored sub-sections with summaries and span boundaries, and a causal graph whose directed edges record how entities influence one another across the whole document. Each tree node that needs editing carries its own modification suggestion; nodes that need no change are simply omitted, so untouched regions stay verbatim. The final revision is generated from these tree-structured suggestions with the causal graph consulted to extend every change through the entity relations it triggers. The paper's experiments claim that this pipeline consistently beats direct prompting on faithfulness, logical coherence, and fluency, with the causal graph as the larger contributor on logically dense texts.","pith_inferences":["The framework's design implies a cheap diagnostic the paper does not run: extract the causal graph alone, show its edges to a human editor, and ask whether those edges identify the passages a careful editor would deem implicitly relevant; strong agreement would isolate the mechanism behind the win rates.","The same two structures could support iterative editing sessions: rebuild the summary tree and causal graph after each revision so a user can apply successive changes without the model drifting into over- or under-editing.","Because the dataset is built by asking an LLM to generate modification suggestions for existing QA and summarization texts, benchmark difficulty inherits the generator's skill; adding contradiction-inducing or multi-hop suggestions would stress-test whether the causal graph is actually doing propagation rather than merely being present.","The evaluation aggregates faithfulness, coherence, and fluency into one judge call; scoring the three criteria separately, and measuring how often unedited regions survive byte-for-byte, would directly test the mechanism the framework claims."],"forward_implications":["Direct prompting of an LLM to 'edit this text' is the wrong baseline for long-form editing; the paper argues that pre-structuring the document around its entities and their causal relations is what separates faithful edits from destructive ones.","The two failure modes are separable: the summary tree alone confines edits to relevant regions, and adding the causal graph yields the largest extra gains on texts with dense internal logic, such as narratives, legal documents, and meeting transcripts.","The framework is training-free and its largest net gain appears on the weakest model (GPT-4o-mini, 39.40%), so the value of the approach concentrates precisely where baseline editing quality is lowest.","The paper ships a reusable eight-dataset, multi-domain benchmark for long-form text modification, with metadata-guided modification suggestions, so later methods can be measured against the same pairs.","Ablations attribute most of the win on logically dense texts to the causal graph (an average 20.74-point win-rate drop without it), while weakly structured datasets lose little, suggesting the graph's role varies by input."],"supporting_citations":[{"why":"Supplies the chain-of-thought rationale for asking the model to state each entity's required adjustment before scoring its importance.","marker":"(Wei et al., 2022)"},{"why":"NarrativeQA is one of the seven sources from which the evaluation texts are drawn.","marker":"(Kociský et al., 2018)"},{"why":"QuALITY supplies long fiction passages with QA metadata used to generate modification suggestions.","marker":"(Pang et al., 2022)"},{"why":"GOVREPORT provides the government-report texts in the evaluation dataset.","marker":"(Huang et al., 2021)"},{"why":"LongBench/MultiFieldQA contributes the English and Chinese multi-field texts and metadata.","marker":"(Bai et al., 2024)"},{"why":"MuSiQue contributes the multi-hop QA passages to the dataset.","marker":"(Trivedi et al., 2022)"},{"why":"QASPER contributes research-paper texts anchored to questions and answers.","marker":"(Dasigi et al., 2021)"},{"why":"QMSum provides the meeting-transcript domain and its query metadata.","marker":"(Zhong et al., 2021)"},{"why":"Grounds the evaluation design, including the bias-mitigation discussion for using an LLM as judge.","marker":"(Zheng et al., 2023)"},{"why":"Defines GPT-4o and GPT-4o-mini, the primary baselines and the judge model.","marker":"(Hurst et al., 2024)"}],"fun_headline_variants":["Hierarchical-causal edits boost long-text win rates to 79%","Causal graph + summary tree lift editing wins to 79.5%","Tree and causal maps refine long-form edits, 79% win rate","HiCaM: causal structure beats prompts in text editing","Summary trees and causal graphs fix long-text edits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every quality judgment in the experiments comes from a single AI judge, GPT-4o, comparing pairs of outputs, and the paper reports no check of whether human readers would agree with that judge's calls.","fun_headline_variants_meta":{"raw":{"variants":["Hierarchical-causal edits boost long-text win rates to 79%","Causal graph + summary tree lift editing wins to 79.5%","Tree and causal maps refine long-form edits, 79% win rate","HiCaM: causal structure beats prompts in text editing","Summary trees and causal graphs fix long-text edits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000524,"raw_usage":{"total_tokens":2513,"prompt_tokens":907,"completion_tokens":1606,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":1527}},"tokens_in":523,"tokens_out":1606,"duration_ms":11720,"temperature":1.0,"reasoning_tokens":1527,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:26:46.308368+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a human preference study on a few hundred of the benchmark's comparison pairs: with readers blind to which output came from which method, compare their choices with GPT-4o's verdicts—if agreement is near chance, the reported win rates measure the judge's tastes rather than editing quality. A complementary check targets the mechanism directly: because HiCaM promises to leave irrelevant spans untouched, measure the fraction of unedited regions kept byte-for-byte in HiCaM outputs versus direct-prompt outputs.","supporting_citations":[],"review_version":1}