{"id":"2f097265-b241-464f-902b-746fa2021d53","arxiv_id":"2608.04569","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Independent hard prompt compression frequently retains an answer while deleting the definition or bridge needed to interpret it, and fixed-budget reinsertion of that missing support substantially improves QA accuracy.","lead":"Hard prompt compression methods, which keep only the highest-scoring text fragments under a token budget, often delete the definition or bridge sentence needed to interpret the text they keep; the paper calls this referential dangling. Across three multi-hop QA datasets at 30% compression, 34-54% of bridge examples with a retained answer paragraph lose the needed definition paragraph, and restoring it within the same budget improves answer accuracy by 29-34 points.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline prevalence rates rest on the Section 4 content-word overlap proxy for Eq. (1); a 40-example audit and a threshold sweep do not fully anchor the 34-54%, 32-60%, and 'every document' claims to exact referential completeness.","rationale":"The reader's weakest assumption is also the most load-bearing one I can identify. The paper's title-level conclusion that independent hard compression has a paradigm-level referential-dangling failure, and the abstract's quantitative claims (34-54%, up to 60%, every LongBench-v2 document), all rest on the content-word overlap proxy defined in Section 4. Proposition 1 establishes the existence of the failure mode under the additive model without any measurement, but existence alone does not support the prevalence claims. The human audit of 40 examples and the threshold sweep in Appendix A are genuinely helpful: the audit shows that flagged examples are usually genuinely dangling (95% precision), and the threshold sweep shows the ordering is not an artifact of the 0.5 cutoff. However, precision on 40 flagged examples says nothing about false negatives, and the LongBench-v2 diagnostic is not clearly tied to question relevance. The reselection experiment is better controlled because it uses annotated supporting paragraphs and holds the budget fixed, but it is run only on Beaver outputs and its dangling subsets are selected by the same proxy. The automatic-restoration result is honestly reported as not significantly beating random insertion and not transferring significantly across compressors, so I do not treat that as a reason to reject. I see no internal inconsistency in the formal framework; the additive model and Proposition 1 are sound. The code and trained model are released, and the statistical tests are appropriate. If the exact-retention check matches the proxy rates, the paper should move toward acceptance; if not, the strong prevalence wording must be revised. Since the reader already assigned CONDITIONAL for essentially this reason, my recommendation is unchanged.","tokens_in":19211,"tokens_out":3575,"duration_ms":42408,"concrete_test":"On the exact HotpotQA Beaver sample used for Figure 2a (n=300 at r=0.30), compute Eq. (1) exactly: for each bridge question, use HotpotQA's supporting-fact annotations to determine whether the retained text contains the full answer sentence and the full definition/bridge sentence needed to interpret it, with two independent annotators resolving cases where annotations are incomplete. Compare the exact answer-path dangling rate and complete-evidence retention rate with the proxy-based 34.2% and 61.0% values. If exact rates lie within 5 percentage points, the proxy concern is settled; if they fall outside that band, the headline prevalence must be restated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that referential dangling is a systematic, paradigm-level failure depends on measured prevalence. Section 4 replaces the exact support-set condition of Eq. (1) with directional content-word overlap cov(v,A) >= 0.5: an answer unit is 'retained' if half its content words appear in any output unit, and a definition unit is 'missing' when the same predicate fails. Table 1, Figure 3, and the LongBench-v2 'every document' result all inherit this proxy. The paper validates the proxy on only 40 human-audited flagged examples (38 required, 2 false positives). That bounds false positives in a small flagged sample, but it does not measure false negatives and does not validate the sentence-level cross-compressor events or the LongBench-v2 diagnostic. The LongBench-v2 first-mention diagnostic is especially loose: it counts any later reference whose first-mention sentence is omitted, whether or not that reference is needed for the question, so 'every document contains at least one dangling reference' can overstate Eq. (1). Appendix A sweeps the overlap threshold from 0.3 to 0.7 and shows that dangling remains substantial under that proxy; it does not show that the proxy tracks actual interpretability. If exact retention on the same examples yields materially lower rates, the headline becomes a small existence result (already provided by Proposition 1) rather than a measured systemic failure. The fixed-budget reselection experiment is stronger because it uses annotated supporting paragraphs directly, but it operates on dangling subsets defined by the same proxy, so its scope inherits the same uncertainty.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies a failure mode of hard prompt compression: independent scoring can retain a relevant fragment while deleting a dependency needed to interpret it, which it calls referential dangling. It formalizes the notion in Section 3, proves an existence result (Proposition 1), and then measures prevalence using a directional content-word overlap proxy. The reported empirical claims are that Beaver at compression ratio 0.30 leaves the answer path incomplete in 34-54% of bridge examples across HotpotQA, 2WikiMultiHopQA, and MuSiQue; that all six hard compressors tested exhibit dangling rates from 32% to 60% on a shared HotpotQA bridge set; that every LongBench-v2 Single-Document QA document contains at least one dangling reference; that fixed-budget reselection improves Qwen3-8B accuracy by 29-34 points on dangling examples; and that a trained classifier can restore some of the loss automatically, improving accuracy by 4.7 points on HotpotQA while changing the compression ratio from 0.30 to 0.31. The paper concludes that hard compressors should optimize both relevance and referential completeness.","tokens_in":19424,"tokens_out":6531,"duration_ms":70322,"significance":"If the prevalence estimates hold, this is an important and actionable observation: it identifies a structural failure shared across scoring signals, and the fixed-budget accuracy gains suggest that the phenomenon has practical cost. The paper has genuine strengths: a clear formal definition with a self-contained existence proof, a controlled fixed-budget comparison with exact McNemar tests, robustness sweeps over the overlap threshold and compression ratio, exact model identifiers in Table 9, and released code, results, and trained models. However, the central prevalence claims rest on a proxy that is only partially validated, and the main intervention changes multiple aspects of the context simultaneously. These issues are load-bearing because the paper's headline contribution is that referential dangling is a systematic, paradigm-level failure rather than a rare artifact.","major_comments":[{"comment":"The headline prevalence rates in Table 1, Figure 3, and the LongBench-v2 result are not direct measurements of the sufficiency condition in Eq. (1). They rely on directional content-word overlap with threshold 0.5 and, for the bridge datasets, check only the paired definition paragraph without enumerating alternative support paths that Eq. (1) explicitly allows. The manuscript acknowledges both limitations ('This protocol approximates Equation (1) rather than evaluating it exactly' and 'This test does not enumerate alternative support paths'). The 40-example human audit in Appendix B estimates precision on flagged cases only (38/40); it does not measure false negatives among unflagged examples, and the Appendix A threshold sweep shows stability of the proxy, not that the proxy tracks exact referential completeness. Because the central claim that dangling is systematic is a prevalence claim, please add a random-sample audit that applies exact sentence retention or human sufficiency judgments to both flagged and unflagged examples and reports precision and recall, or explicitly downgrade the prevalence claims to proxy-based estimates.","section":"Section 4, Measurement protocol"},{"comment":"The first-mention diagnostic counts any retained later reference whose first-mention sentence is absent, regardless of whether that reference is needed to answer the query. This does not instantiate Eq. (1), which requires the dangling sentence to be in T_{C,Q}. Consequently the statements 'every document in LongBench-v2 Single-Document QA contains at least one dangling reference' and the mean rate of 30.5% are not evidence for the paper's query-dependent notion of referential dangling. Please restrict the diagnostic to question-relevant entities or references, or relabel these numbers as 'omitted first mentions' rather than 'dangling references' in the abstract and conclusion.","section":"Section 4.1, Long documents"},{"comment":"Panel 1 of Table 3 shows large gains for Reselected over Base, but the intervention changes two things at once: it reinserts the missing supporting paragraph and removes the lowest-scoring nonsupporting paragraphs. The paper acknowledges this ('these comparisons measure the joint change in retained content rather than the isolated contribution of the reinserted paragraph'). Without an ablation that removes nonsupporting paragraphs without reinsertion, or reinserts with a budget increase, the experiment does not establish that referential completeness, rather than distractor removal, drives the 28.8-34.3 point gains. The matched addition control in Appendix G (+2.0 points for random insertion versus +4.7 for targeted restoration) is suggestive but the direct contrast is not significant (p=0.15). Please add the missing ablation or weaken the causal conclusion about the role of completing the missing support.","section":"Section 5, Comparison protocol"}],"minor_comments":[{"comment":"The abstract says Beaver leaves the answer path incomplete in 34-54% of bridge examples, but Table 1's rho_d is conditional on the answer paragraph satisfying Keep_0.5; please state the denominator explicitly as 'bridge examples with a retained answer paragraph' to avoid overstating the proportion of all bridge examples.","section":"Abstract and Section 4.1"},{"comment":"The main K=3 result was selected after sweeping K, and the reported p=0.022 does not account for this selection. Please report the K sweep with a multiple-comparison caveat or state that K=3 was chosen by a held-out criterion.","section":"Section 6 and Appendix F"},{"comment":"The McNemar tests are described as exact and two-sided in Appendix D; please state this in the main text near Table 3, since the p-values are central to the fixed-budget claim.","section":"Appendix D"},{"comment":"The DAC proxy row in Table 9 lists Qwen3-0.6B as both the scoring and proxy model; a brief sentence explaining the role of the proxy would improve reproducibility.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is serious and the accuracy experiments are valuable, and the authors are transparent about most limitations. My main concern is that the headline prevalence numbers rest on a proxy whose calibration is only partially validated, and the largest intervention changes two factors jointly; both issues are fixable with additional experiments or weaker claims. I recommend major revision rather than rejection because the formal framework and the controlled accuracy comparisons are sound contributions, and the requested validation is within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth a serious look. It identifies a clean failure mode in hard prompt compression: independent scoring can retain the answer sentence while deleting the definition or bridge sentence needed to interpret it. That's a real blind spot, and they show it consistently across six compressors and three multi-hop QA datasets. The fixed-budget reselection experiment is the strongest part: restoring the missing supporting paragraph recovers 29–34 points of accuracy at no extra token cost. That's a credible, controlled result with McNemar significance.\n\nWhat's actually new: the formal definition of referential dangling (Equation 1), the simple Proposition 1 existence result, and the systematic measurement. The connection to Durrett et al.'s anaphoricity constraints is fair. The paper ships code and data, which matters.\n\nThe soft spots are real but not fatal. The headline prevalence rates rest on a content-word overlap proxy for the formal condition, validated on only 40 flagged examples. That bounds false positives but not false negatives, and the LongBench-v2 first-mention diagnostic counts any omitted first mention regardless of whether the question needs it. So the specific numbers——34–54%, 32–60%, 'every document'——should not be taken as exact measurements of Equation (1). The authors admit the proxy approximates rather than evaluates exactly, but the abstract doesn't carry the caveat. Appendix A's threshold sweep helps; it shows the pattern is not an artifact of one cutoff.\n\nThe automatic restoration part is weaker. The +4.7 point gain over base is significant, but the matched random-insertion control gives +2.0 and the direct contrast is not significant (p=0.15). The transfer across compressors is also not significant. So the paper's prescriptive contribution is preliminary; the diagnostic contribution is the core.\n\nAlso, the 'paradigm-level' framing outruns the evidence. The paper shows that hard compressors frequently drop support and that fixing it helps; that's a solid within-subfield finding, not a collapse of the paradigm. The authors mostly acknowledge this in the conclusion, so it's a framing issue, not a hidden flaw.\n\nWho this is for: anyone working on prompt compression, long-context inference, or extractive selection. It deserves a serious referee. The referee should push for a better retention metric, a question-relevance filter on LongBench-v2, and a stronger restoration control before publication.\n\nMy take: conditional accept at a good venue after those revisions.","headline":"A real and well-documented failure mode in hard prompt compression, though the headline prevalence numbers rest on a lightly validated proxy and the automatic restoration does not yet beat random insertion.","tokens_in":20072,"tokens_out":2434,"would_cite":true,"duration_ms":21482,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hard prompt compression can keep a relevant answer while deleting the definition that makes it interpretable; the paper shows this 'referential dangling' is systematic and that fixed-budget reselection recovers most of the accuracy.","keywords":["hard prompt compression","referential dangling","multi-hop question answering","context compression","large language models","referential completeness","dependency preservation","answer-path incompleteness"],"falsifier":"Take a random sample of several hundred flagged bridge examples, have independent annotators apply the paper's formal sufficiency condition (Equation 1) to decide whether the omitted paragraph is genuinely required to interpret the retained answer, and recompute the rates; if the true rate is near zero, or if a fresh fixed-budget reselection on new bridge data recovers well under 88% of the gap to full support, the central claim fails.","tokens_in":18919,"feed_emoji":"🧩","tokens_out":18066,"duration_ms":154907,"temperature":0.7,"pith_summary":"The paper tries to establish that hard prompt compression—scoring tokens, sentences, or chunks independently and keeping the highest-scoring units under a token budget—has a structural failure: it can retain a relevant answer while deleting the definition or bridge fact needed to interpret it. The paper calls this referential dangling and argues it is recurrent, not incidental: at compression ratio 0.30, Beaver leaves the answer path incomplete in 34-54% of bridge examples across HotpotQA, 2WikiMultiHopQA, and MuSiQue; all six hard compressors tested dangle on 32-60% of a shared HotpotQA bridge set; and every LongBench-v2 Single-Document QA document in the sample has at least one dangling reference. On dangling examples, reinserting the omitted supporting paragraph within the same token budget improves Qwen3-8B accuracy by 29-34 percentage points ($p < 0.0001$), recovering at least 88% of the gap to contexts that keep both supporting paragraphs, and stronger answer models do not absorb the loss. A compact classifier that ranks omitted sentences by whether they are needed to interpret retained text reinserts candidates automatically and adds 4.7 points on HotpotQA while the compression ratio moves only from 0.30 to 0.31. The paper concludes that hard compressors should optimize both relevance and referential completeness.","feed_headline":"34-54% of compressed multi-hop answer paths go incomplete","feed_subtitle":"At 70% compression, six selectors dangle on 32-60% of bridge cases; restoring one paragraph recovers 29-34 points.","key_machinery":"The central object is the formal condition for referential dangling, built on inclusion-minimal sufficient support sets in the spirit of extractive summarization. For an extractive compressed output, $\\operatorname{Ret}_C(A)$ is the set of source sentences fully retained; $T_{C,Q}$ is the set of task-relevant sentences; and $\\mathcal{D}_{C,Q}(s)$ is the family of inclusion-minimal subsets of other sentences that, together with $s$ and the question, contain all explicit information needed to interpret and use $s$. Equation (1) says the output dangles when some retained task-relevant sentence has no fully retained sufficient support set. The matching machinery is the additive selection model: each fragment gets a scalar utility, and the selector maximizes summed utility under a token budget with no joint-retention term; Proposition 1 shows this guarantee is absent even when a complete alternative fits the budget. The empirical proxy used to apply the condition to real outputs is directional content-word overlap $\\operatorname{cov}(v,A) \\ge 0.5$ between an output unit and a source unit.","core_discovery":"The paper's central claim is that independent scoring cannot guarantee that a compressed prompt remains self-contained: the additive selection objective has no interaction term enforcing joint retention of a fragment and the text that interprets it. Formally, a compressed context exhibits referential dangling when it retains a task-relevant sentence $s$ but retains no member of $\\mathcal{D}_{C,Q}(s)$, the family of inclusion-minimal sufficient support sets for $s$ (Equation 1). Proposition 1 shows that for three equal-cost fragments $\\{x,d,z\\}$ and budget $2m$, the unique additive maximizer can be $\\{x,z\\}$, whose output dangles, even though $\\{x,d\\}$ is feasible and complete. The paper then measures the phenomenon across implementations: at ratio 0.30, Beaver dangles 34.2-54.2% of bridge examples across three datasets; six compressors score 32.1-59.8% on a shared HotpotQA bridge set; and every LongBench-v2 document has at least one dangling reference. Fixed-budget reselection on dangling examples improves Qwen3-8B accuracy by 28.8-34.3 points and recovers at least 88% of the gap to full support, while GPT-5.5 remains 8.8 points less accurate on compressed MuSiQue contexts.","pith_inferences":["The same split-evidence failure should appear in retrieval-augmented generation, where scored or truncated passages can keep an answer string while dropping the passage that defines a referenced entity; the LongBench-v2 first-mention result is early evidence of that wider reach.","A direct design implication, not built in the paper, is to add a joint-retention bonus to the selection objective—score a fragment together with the text that interprets it—instead of restoring omitted sentences as a separate pass.","The automatic-restoration gain is probably limited by candidate construction rather than by the classifier, because a diagnostic condition that put the annotated supporting sentence in the candidate set recovered 8.0 points versus 4.7 for first-mention candidates.","Reported prevalence should be read as proxy measurements: a human-annotated application of the formal condition could shift the exact rates, especially the every-document LongBench-v2 result."],"forward_implications":["At compression ratio 0.30, a query-aware chunk compressor leaves the answer path incomplete in 34-54% of bridge examples across HotpotQA, 2WikiMultiHopQA, and MuSiQue.","Across six hard compressors using different scoring signals, the dangling rate on a shared HotpotQA bridge set ranges from 32.1% to 59.8%, so the failure is not tied to one selection method.","Every one of the 80 LongBench-v2 Single-Document QA documents contains at least one dangling reference under the first-mention diagnostic, with rates from 25% to 37% across seven subdomains.","Reinserting the missing supporting paragraph while removing nonsupporting paragraphs at the same token budget improves Qwen3-8B accuracy by 28.8-34.3 points on dangling examples and recovers at least 88% of the gap to retaining both supporting paragraphs.","Automatic restoration with a small classifier adds 4.7 points on HotpotQA with Qwen3-8B while the compression ratio changes only from 0.30 to 0.31."],"supporting_citations":[{"why":"Defines Beaver, the query-aware hierarchical chunk compressor whose outputs drive the prevalence, reselection, and restoration experiments.","marker":"Hu et al. 2026"},{"why":"Supplies HotpotQA bridge questions with supporting-paragraph annotations, the main dataset for measuring dangling and for the restoration evaluation.","marker":"Yang et al. 2018"},{"why":"Supplies the compositional 2WikiMultiHopQA examples used to show the 34-54% prevalence range.","marker":"Ho et al. 2020"},{"why":"Supplies MuSiQue, where dangling is most frequent and where stronger downstream models still lose accuracy under compression.","marker":"Trivedi et al. 2022"},{"why":"Supplies LongBench-v2 Single-Document QA, used for the first-mention diagnostic that flags every document.","marker":"Bai et al. 2025"},{"why":"Provides the anaphoricity/referential-completeness constraint that the paper's formal definition extends and that hard compressors lack.","marker":"Durrett, Berg-Kirkpatrick, and Klein 2016"},{"why":"Provides LLMLingua-2, the token-level trained-classifier compressor that dangles in the six-way comparison.","marker":"Pan et al. 2024"},{"why":"Provides LongLLMLingua, the query-aware perplexity compressor with the highest dangling rate in the shared-set comparison.","marker":"Jiang et al. 2024"},{"why":"Identifies Qwen3-8B, the downstream model used for the reselection and automatic-restoration accuracy measurements.","marker":"Yang et al. 2025"},{"why":"Identifies GPT-5.5, the proprietary model used to show that stronger answerers still lose 8.8 points on compressed MuSiQue contexts.","marker":"OpenAI 2026"}],"fun_headline_variants":["Compressed prompts dangle: 34-54% lose answer paths","Referential dangling: hidden flaw in hard prompt compression","Hard compression cuts answer link, hurting accuracy by up to 34 pts","Six compressors dangle on up to 60% of bridge queries","Restoring one paragraph fixes 29-34 pts of compressed accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that 50% content-word overlap between a retained unit and the original supporting text really means the support is present; if that proxy is too lenient or too strict, the dangling rates and the every-document LongBench-v2 result change.","fun_headline_variants_meta":{"raw":{"variants":["Compressed prompts dangle: 34-54% lose answer paths","Referential dangling: hidden flaw in hard prompt compression","Hard compression cuts answer link, hurting accuracy by up to 34 pts","Six compressors dangle on up to 60% of bridge queries","Restoring one paragraph fixes 29-34 pts of compressed accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1595,"prompt_tokens":1169,"completion_tokens":426,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":785,"completion_tokens_details":{"reasoning_tokens":335}},"tokens_in":785,"tokens_out":426,"duration_ms":4585,"temperature":1.0,"reasoning_tokens":335,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:49:57.555494+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of several hundred flagged bridge examples, have independent annotators apply the paper's formal sufficiency condition (Equation 1) to decide whether the omitted paragraph is genuinely required to interpret the retained answer, and recompute the rates; if the true rate is near zero, or if a fresh fixed-budget reselection on new bridge data recovers well under 88% of the gap to full support, the central claim fails.","supporting_citations":[],"review_version":1}