Measuring Alignment With Reader Highlights Net of Position and Length

Kazuki Nakayashiki; Keisuke Watanabe

REVIEW 2 major objections 4 minor 1 cited by

Language-model rankings retain crowd-marked sentences at roughly twice the rate of the best classical extractive baseline, net of position and length.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:12 UTC pith:Y6VNX2JT

load-bearing objection A serious, unusually transparent measurement paper whose headline survives most of its own stress tests, but the contamination gap and tie-break sensitivity keep the result from being definitive. the 2 major comments →

arxiv 2607.27739 v1 pith:Y6VNX2JT submitted 2026-07-30 cs.IR cs.CLcs.HC

Measuring Alignment With Reader Highlights Net of Position and Length

Kazuki Nakayashiki , Keisuke Watanabe This is my paper

classification cs.IR cs.CLcs.HC

keywords social highlightingcrowd-marked sentencesposition biaslength biaslanguage model rankingconfound controlcontext compression evaluation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Context compression is usually judged by another model's downstream accuracy, which makes the evaluation circular. The paper uses naturalistic social highlighting — many readers independently marking the same web page — as a non-circular reference, and asks whether a language-model importance ranking keeps the same sentences readers kept. The central result is that after matching each crowd-marked sentence to unmarked neighbours at equal relative depth and equal within-document length rank, a language-model ranking keeps 38.4% of marked sentences against 19.9% of matched neighbours, an enrichment of +0.196 (95% CI [+0.148, +0.239], exact randomization p=0.0005), replicated across two vendors. The paper further shows this estimator's own false-positive rate is 4.5–7% on synthetic nulls, whereas depth-only stratification — the control one would reach for first — fires on 20–36% of nulls containing no effect. It also reports that on the same budget, a single human reader scores +0.182, indistinguishable from the language model's +0.184.

Core claim

On the paper's own terms, the discovery is that readers' unprompted highlighting is substantially predictable by an off-the-shelf language model once the two dominant confounds — sentence position and sentence length — are removed by within-document matching. The headline number is +0.196 enrichment, with the model keeping 38.4% of crowd-marked sentences against 19.9% of matched unmarked neighbours. The same estimator gives classical word-frequency heuristics +0.088 and lexical centrality +0.098, so the model doubles rather than categorically exceeds cheap lexical selection. Scored identically, a single human reader reaches +0.182, and the paired model-minus-human difference is +0.002 (CI sp

What carries the argument

The argument is carried by a matched-enrichment estimator: for each crowd-marked sentence, the estimator finds unmarked sentences in the same document within 0.05 relative depth and 0.05 within-document length rank, subtracts the compressor's keep rate on those neighbours from its keep rate on the marked sentence, and aggregates across documents with domain-clustered bootstrap intervals. The companion mechanism is calibration on synthetic nulls — keep sets generated from position and length alone over the real corpus geometry — which turns the control's error rate into a measured quantity rather than an assumption. Matching discards 37.9% of marked sentences that lack an admissible comparato

Load-bearing premise

The crowd label is the top 15% of sentences by mark count with at least two marks, and because roughly two-thirds of the label slots sit exactly at the cutoff, the seeded-jitter tie-break decides much of the label — if that tie-breaking were meaningfully different, the headline enrichment could shift outside the reported range.

What would settle it

Reproduce the label using a tie-break-independent rule (for example, a continuous mark count without thresholding) and recompute the matched enrichment; if it falls below the null's 97.5th percentile (+0.054), the headline is an artifact of the label construction. A simpler check: on the same 120 documents, shuffle mark assignments within depth/length strata and confirm the estimator returns enrichment near zero — if it exceeds the calibrated threshold, the control is still leaking.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Compression systems can be evaluated against reader behaviour instead of only downstream task accuracy, giving a non-circular yardstick for what a compressor preserves.
The language-model advantage over classical lexical selection is about a factor of two, so cheap word-counting methods are not null controls; an evaluation that pits a model only against a random baseline overstates the gain.
Any claim of model-human alignment on highlights must control for position and length; depth-only stratification is shown to be an unsafe control, and matching tolerance plus label contamination can move weak contrasts materially.
On the same budget, an off-the-shelf language model is statistically indistinguishable from one member of the crowd it is predicting, which gives the +0.196 number a human-scale reference.
The headline is configuration-sensitive: at compression ratio r=0.05 the effect is 42% of the r=0.20 headline, and prompt framing changes it by a factor of 2.1.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the matched enrichment survives on a broader, non-platform corpus, social highlighting could become a continuous, low-cost benchmark for prompt compression; the platform's convenience sample is the main external-validity limit.
Because two-thirds of the label slots at q=0.15 sit at the cutoff and the production tie-break seed is not available, the exact headline (+0.196) is not reproducible from the artifact; future label definitions should either use a continuous mark count or report a tie-break-insensitive interval.
The paper's explicit non-replication of its own earlier finding suggests that previously reported 'weak' model-human salience correlations may partly reflect uncontrolled position and length, and re-running existing datasets with a matched estimator could settle the disagreement.
The prompt-framing result — asking what is important to keep beats asking what a reader would highlight — hints that language models have implicit theories of 'importance' that differ from their theories of reader behaviour; that asymmetry is a testable handle on what the models actually learned.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A serious, unusually transparent measurement paper whose headline survives most of its own stress tests, but the contamination gap and tie-break sensitivity keep the result from being definitive.

read the letter

This is a real paper, worth a serious read. The core measurement - language-model importance rankings retain crowd-marked sentences at about twice the best classical baseline, +0.196 against +0.098, p=0.0005 under an exact randomization test, replicated cross-vendor - is supported by a lot of careful work. The paper does something genuinely rare: it calibrates the confound control on synthetic nulls and shows that depth-only stratification false-positives at 20-36%. That is an important methodological warning, and it is now the first thing I would cite from this paper. The single-human-reader baseline, computed on the same estimator with the crowd label recomputed to exclude the reader, is also a thoughtful scale check, and the model-human difference of +0.002 is a striking, credible result. The transparency is not cosmetic. The appendix listing eleven withdrawn claims, including the failed covariate ladder and the lead-calibration false positive, is the kind of honesty that should make referees trust the surviving numbers. The paper also reproduces its own non-replication of prior work. That is exactly the behavior you want in this literature. The soft spots are real but not disqualifying. First, the label tie-break: at q=0.15 two-thirds of the label slots sit at the cut value and are resolved by seeded jitter, and the production seed is not in the artifact. The front-loading alternative gives +0.222, outside the redraw band, so the exact headline is not reproducible from what is shipped. The qualitative conclusion probably survives, but the paper should present the effect as a function of tie-break form, not hide behind one seed. Second, the pretraining contamination gap is genuine. The mark-recency test compares mark recency, not document recency, so a document that entered pretraining years ago but accumulated marks recently lands in the after group. The paper admits this, but the admission does not close the gap. Cross-vendor agreement is exactly what shared web pretraining predicts. The randomization test and bootstrap treat the ranking as fixed, so they cannot distinguish inference from retrieval. That limits the construct being measured: alignment with readers is not fully identified as text-based alignment. Third, the headline ratio r=0.20 sits high in a range where the effect varies by a factor of 2.7; at r=0.05 the effect drops to 42% of the headline. The abstract should not lead with the most favorable point. Still, the central claim - at least as a measurement of association, net of position and length - holds up under the perturbations the authors actually ran. This paper deserves a rigorous referee. I would send it to review, and I would ask the authors to either secure a truly held-out corpus or soften the interpretation to retrieval-robust agreement. The method and the calibration result will be useful regardless of the final interpretation.

Referee Report

2 major / 4 minor

Summary. The paper proposes a matched estimator for measuring whether an LLM's importance ranking aligns with naturalistic crowd highlights, after removing position and length confounds. Each crowd-marked sentence is compared against unmarked sentences at equal relative depth and equal within-document length rank; the estimator is calibrated on synthetic nulls built from position and length only. On 120 web documents, the authors report a matched enrichment of +0.196 (95% CI [+0.148, +0.239]) for GPT-5.4 against a 19.9% keep rate for matched neighbours, with p=0.0005 under an exact within-stratum randomization test, replicated with Claude Opus 5. A single human reader scores +0.182 on the same task, and the model—human paired difference is +0.002 (CI spanning zero). Classical baselines are not null: Luhn's heuristic reaches +0.088 and lexical degree centrality +0.098. The paper also reports extensive robustness checks, a clean-control analysis, a covariate balance table, a prompt-framing contrast, and a list of withdrawn claims from internal review.

Significance. If the identification gap is closed, this is a significant contribution to compression evaluation and to the study of LLM—reader alignment. The paper is unusually transparent: it ships a de-identified per-sentence artifact and runnable code, measures the false-positive rate of its own confound control, reports a non-replication of the authors' prior result, and explicitly bounds residual confounds. The human-reader calibration gives the headline number an interpretable scale. However, the central construct—'alignment with reader highlights'—is threatened by pretraining contamination: the corpus is deliberately drawn from the well-read tail of a public platform whose highlight renderings are crawlable, so a model could retrieve stored highlights rather than infer them from text. The paper's mark-recency test does not close this gap, because mark recency is not document recency. The tie-break non-reproducibility is a second, independent load-bearing issue. These are fixable with additional analyses or re-scoped claims, but they currently prevent full acceptance.

major comments (2)

[§6 (Pretraining contamination)] The mark-recency split does not test the contamination channel the corpus most plausibly exposes: a document may have entered pretraining long ago and accumulated highlights recently, so the 'after' group is not a held-out group. Since the corpus is the well-read tail of a public platform whose per-URL highlight renderings are crawlable, shared web pretraining can produce exactly the cross-vendor agreement the paper cites as evidence of generality. The randomization and bootstrap intervals condition on the ranking as fixed and therefore cannot distinguish inference from retrieval. To keep the central claim, please report a split by document publication date relative to each model's training cutoff (or a post-cutoff subset), or re-scope the conclusion to 'ranking behavior, mechanism unspecified.' Without such a control, the sentence 'an off-the-shelf language model predicts the crowd abou
[§5.3 (Label definition and tie-break)] At q=0.15, 1,407 of 2,095 label slots lie at the cut value and are decided by seeded jitter from the Firestore document identifier; the de-identified artifact deliberately omits that seed. The headline +0.196 cannot be exactly reproduced: the label sweep's +0.186 on 1,301 pairs is one random redraw, and the earlier front-loading tie-break gave +0.222, outside the reported redraw band. Because two-thirds of the label is determined by the tie-break, the exact p-value and enrichment are partly artifacts of an unreleased random stream. Please either include a de-identified deterministic seed (or a reproducible hash) so the exact label can be reconstructed, or report the tie-break ensemble (mean and interval over many redraws) as the headline. The paper's transparency is commendable, but the current presentation overstates the precision of the point estimate.

minor comments (4)

[§4.1, Eq. (1)] The notation ⊮ is nonstandard; please define it explicitly or replace with \(\mathbb{1}\) or an indicator variable.
[§5.2] Typo: 'W e state that as not established' should be 'We state...'.
[References] Reference [9] lists 'Hardy, Shashi Narayan, Andreas Vlachos'; the first author's given name appears to be missing or misformatted.
[§5.3 (Label sweep)] The distinction between the production tie-break stream and the sweep's random tie-break is important and clearly stated, but the row label '+0.186' could be misunderstood as a reproduction of Table 2. Consider adding a footnote or changing the row label to 'random redraw (one draw)'.

Circularity Check

0 steps flagged

No significant circularity: the reference signal is naturalistic and independent of the models, the models are zero-shot and not fitted to the labels, and the paper calibrates and stress-tests its own estimator.

full rationale

The paper's central derivation is not circular. The reference signal is crowd highlighting on public web pages, produced by independent readers and not generated by the models under evaluation. The language-model rankings are zero-shot importance rankings; no parameter is fitted to the highlight labels. The matched estimator removes position and length, and the paper validates it on synthetic nulls in which keep sets are generated from position and length alone, measuring false-positive rates rather than assuming the control works. The single-reader baseline recomputes the crowd label with that reader excluded, avoiding self-prediction. The only self-citation is to the authors' prior work [1], and the paper explicitly reports that a claim from that work does not reproduce on this corpus, so the prior work is not load-bearing. The pretraining-contamination concern is a validity threat, not a circularity in the derivation chain: it is an empirical identification gap, and the paper addresses it with a recency test while acknowledging the test's limits. No equation reduces to its own input, and no fitted parameter is renamed as a prediction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim does not introduce new physical entities or fitted constants. The free parameters are methodological thresholds (label fraction, matching tolerance, compression ratio, null threshold) that are all swept or disclosed. The key domain assumption is that highlights reflect reader choice, and that matching on depth and length removes the relevant confounders; the balance table bounds residual imbalance. No new particles or mechanisms are postulated.

free parameters (4)

label top fraction q = 0.15
Sentences in the top 15% by mark count with ≥2 marks are labelled crowd-marked. Swept over 0.10–0.25 with stable results, but at q=0.15 two-thirds of label slots are at the cut value and decided by tie-break.
matching tolerance = 0.05 (also 0.03, 0.10)
Within 0.05 relative depth and 0.05 within-document length rank. The paper reports a tolerance grid because tightening to 0.03 sharply raises the discard rate; the choice affects the estimate (+0.160 to +0.218).
compression ratio r = 0.20 (headline)
The pre-registered set included 0.10–0.50; the sweep shows the effect varies by a factor of 2.7 (from +0.083 at 0.05 to +0.222 at 0.30), and 0.20 sits high in the range.
null threshold reference = +0.054
The null's 97.5th percentile, used as a magnitude reference. The paper explicitly clarifies it is approximately two standard errors, not a confound floor.

axioms (4)

domain assumption Social highlights are an unprompted record of what readers chose to keep
The paper states this is true by construction of the measurement and explicitly disclaims that highlights mark objective importance (§1).
domain assumption Within matched depth/length bins, marked versus unmarked assignment is exchangeable under the null
Licenses the exact randomization test; the balance table bounds residual imbalance but cannot rule out unmeasured confounders (§4.1, §5.2).
ad hoc to paper Synthetic nulls built from position/length-only keep rules over real corpus geometry represent the null distribution
The calibration is sensitive to the strength of the synthetic confound, so the paper reports four columns in Table 1; this is an assumption about the null-generating process (§4.2).
standard math Domain-clustered bootstrap and randomization test assume independence across domains
Used for intervals and p-values; the paper clusters by domain (71 domains) to account for within-domain correlation (§4.1).

pith-pipeline@v1.3.0-daily-deepseek · 11672 in / 10185 out tokens · 100626 ms · 2026-08-01T02:12:55.715564+00:00 · methodology

0 comments

read the original abstract

Context compression discards most of a document before a language model reads it, and is normally evaluated by downstream task accuracy - which makes another model the judge of what mattered. Naturalistic social highlighting offers a non-circular reference: many people independently marking passages on the same page. But the obvious metric, the fraction of crowd-marked sentences a compressor keeps, is confounded twice: crowd marks are front-loaded and crowd-marked sentences are longer, so any method favouring early or long sentences scores well regardless of readers. We remove both by matching each marked sentence against unmarked sentences of the same document at equal relative depth and equal within-document length rank, and we calibrate every estimator on synthetic nulls built from position and length alone - a step that matters, since depth-only stratification returns a false positive on 20-36% of nulls containing no effect. On 120 web documents (at least 12 independent readers each), a language-model importance ranking keeps 38.4% of crowd-marked sentences against 19.9% of their matched neighbours: an enrichment of +0.196 [+0.148, +0.239], at p = 0.0005 under an exact randomization test that assumes nothing about clustering, and replicated cross-vendor. Naive truncation, whose keep rule is position, correctly falls to +0.003. To give the number a scale: scored identically, on the same budget, against a crowd label recomputed to exclude them, a single human reader reaches +0.182 - indistinguishable from GPT-5.4 (+0.002 [-0.081, +0.088]) and below Claude Opus 5. Classical methods are not null - Luhn's 1958 heuristic reaches +0.088 - so reader selection is partly recoverable by counting words; conditioning additionally on lexical centrality removes only 0.010, so the agreement is not centrality. We also report that a claim in our own prior work does not reproduce on this corpus.

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Language Models Agree With Each Other, Not With Readers
cs.IR 2026-07 accept novelty 7.0

Across 18 model arms, model-model excess agreement (+0.093 median) is 2.3x human-human agreement (+0.040), against a naturalistic uninstructed reader baseline.

Reference graph

Works this paper leans on

23 extracted references · 2 linked inside Pith · cited by 1 Pith paper

[1]

Selection, Not Salience: The Shape and Limits of Personalization in Social Highlighting

Kazuki Nakayashiki, Keisuke Watanabe. Selection, Not Salience: The Shape and Limits of Personalization in Social Highlighting. arXiv:2606.10398, 2026

Pith/arXiv arXiv 2026
[2]

A Diagnostic Study of Explainability Techniques for Text Classification.EMNLP, 2020

Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle Augenstein. A Diagnostic Study of Explainability Techniques for Text Classification.EMNLP, 2020

2020
[3]

You had me at hello: How phrasing affects memorability.ACL, 2012

Cristian Danescu-Niculescu-Mizil, Justin Cheng, Jon Kleinberg, Lillian Lee. You had me at hello: How phrasing affects memorability.ACL, 2012

2012
[4]

You are no Jack Kennedy

Chenhao Tan, Hao Peng, Noah A. Smith. “You are no Jack Kennedy”: On Media Selection of Highlights from Presidential Debates.WWW, 2018

2018
[5]

LAST at CMCL 2021 Shared Task.CMCL, 2021

Yves Bestgen. LAST at CMCL 2021 Shared Task.CMCL, 2021. 12

2021
[6]

ERASER: A Benchmark to Evaluate Rationalized NLP Models.ACL, 2020

Jay DeYoung et al. ERASER: A Benchmark to Evaluate Rationalized NLP Models.ACL, 2020

2020
[7]

arXiv:2506.05167, 2025

ECoRAG: Evidentiality-guided Compression for Retrieval-Augmented Generation. arXiv:2506.05167, 2025

Pith/arXiv arXiv 2025
[8]

Countering the Effects of Lead Bias in News Summarization.EMNLP-IJCNLP, 2019

Matt Grenander, Yue Dong, Jackie Chi Kit Cheung, Annie Louis. Countering the Effects of Lead Bias in News Summarization.EMNLP-IJCNLP, 2019

2019
[9]

HighRES: Highlight-based Reference-less Evaluation of Summarization.ACL, 2019

Hardy, Shashi Narayan, Andreas Vlachos. HighRES: Highlight-based Reference-less Evaluation of Summarization.ACL, 2019

2019
[10]

Relative Importance in Sentence Processing.ACL, 2021

Nora Hollenstein, Lisa Beinborn. Relative Importance in Sentence Processing.ACL, 2021

2021
[11]

Towards Faithfully Interpretable NLP Systems.ACL, 2020

Alon Jacovi, Yoav Goldberg. Towards Faithfully Interpretable NLP Systems.ACL, 2020

2020
[12]

LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models.EMNLP, 2023

Huiqiang Jiang et al. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models.EMNLP, 2023

2023
[13]

Earlier Isn’t Always Better

Taehee Jung, Dongyeop Kang, Lucas Mentch, Eduard Hovy. Earlier Isn’t Always Better. EMNLP-IJCNLP, 2019

2019
[14]

Content Selection in Deep Learning Models of Summarization.EMNLP, 2018

Chris Kedzie, Kathleen McKeown, Hal Daum´ e III. Content Selection in Deep Learning Models of Summarization.EMNLP, 2018

2018
[15]

A Trainable Document Summarizer.SIGIR, 1995

Julian Kupiec, Jan Pedersen, Francine Chen. A Trainable Document Summarizer.SIGIR, 1995

1995
[16]

Understanding and Improving Information Preservation in Prompt Compression.Findings of EMNLP, 2025

Weronika Lajewska et al. Understanding and Improving Information Preservation in Prompt Compression.Findings of EMNLP, 2025

2025
[17]

Compressing Context to Enhance Inference Efficiency of Large Language Models.EMNLP, 2023

Yucheng Li et al. Compressing Context to Enhance Inference Efficiency of Large Language Models.EMNLP, 2023

2023
[18]

A Cross-lingual Comparison of Human and Model Relative Word Importance.CLASP, 2022

Felix Morger, Stephanie Brandl, Lisa Beinborn, Nora Hollenstein. A Cross-lingual Comparison of Human and Model Relative Word Importance.CLASP, 2022

2022
[19]

LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression.Findings of ACL, 2024

Zhuoshi Pan et al. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression.Findings of ACL, 2024

2024
[20]

How to Compare Summarizers without Target Length?NeuralGen@NAACL, 2019

Simeng Sun, Ori Shapira, Ido Dagan, Ani Nenkova. How to Compare Summarizers without Target Length?NeuralGen@NAACL, 2019

2019
[21]

Behavioral Analysis of Infor- mation Salience in Large Language Models.Findings of ACL, 2025

Jan Trienes, J¨ org Schl¨ otterer, Junyi Jessy Li, Christin Seifert. Behavioral Analysis of Infor- mation Salience in Large Language Models.Findings of ACL, 2025

2025
[22]

arXiv:2602.00459,Findings of EACL, 2026

What Matters to an LLM? Analysing Information Salience in Large Language Models. arXiv:2602.00459,Findings of EACL, 2026

arXiv 2026
[23]

Searching for Effective Neural Extractive Summarization.ACL, 2019

Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, Xuanjing Huang. Searching for Effective Neural Extractive Summarization.ACL, 2019. 13 Appendix A: Claims withdrawn during this work This work was developed against ten rounds of adversarial internal review. Eleven claims were withdrawn, six of them after we had written them up. We list them because the pa...

2019

[1] [1]

Selection, Not Salience: The Shape and Limits of Personalization in Social Highlighting

Kazuki Nakayashiki, Keisuke Watanabe. Selection, Not Salience: The Shape and Limits of Personalization in Social Highlighting. arXiv:2606.10398, 2026

Pith/arXiv arXiv 2026

[2] [2]

A Diagnostic Study of Explainability Techniques for Text Classification.EMNLP, 2020

Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle Augenstein. A Diagnostic Study of Explainability Techniques for Text Classification.EMNLP, 2020

2020

[3] [3]

You had me at hello: How phrasing affects memorability.ACL, 2012

Cristian Danescu-Niculescu-Mizil, Justin Cheng, Jon Kleinberg, Lillian Lee. You had me at hello: How phrasing affects memorability.ACL, 2012

2012

[4] [4]

You are no Jack Kennedy

Chenhao Tan, Hao Peng, Noah A. Smith. “You are no Jack Kennedy”: On Media Selection of Highlights from Presidential Debates.WWW, 2018

2018

[5] [5]

LAST at CMCL 2021 Shared Task.CMCL, 2021

Yves Bestgen. LAST at CMCL 2021 Shared Task.CMCL, 2021. 12

2021

[6] [6]

ERASER: A Benchmark to Evaluate Rationalized NLP Models.ACL, 2020

Jay DeYoung et al. ERASER: A Benchmark to Evaluate Rationalized NLP Models.ACL, 2020

2020

[7] [7]

arXiv:2506.05167, 2025

ECoRAG: Evidentiality-guided Compression for Retrieval-Augmented Generation. arXiv:2506.05167, 2025

Pith/arXiv arXiv 2025

[8] [8]

Countering the Effects of Lead Bias in News Summarization.EMNLP-IJCNLP, 2019

Matt Grenander, Yue Dong, Jackie Chi Kit Cheung, Annie Louis. Countering the Effects of Lead Bias in News Summarization.EMNLP-IJCNLP, 2019

2019

[9] [9]

HighRES: Highlight-based Reference-less Evaluation of Summarization.ACL, 2019

Hardy, Shashi Narayan, Andreas Vlachos. HighRES: Highlight-based Reference-less Evaluation of Summarization.ACL, 2019

2019

[10] [10]

Relative Importance in Sentence Processing.ACL, 2021

Nora Hollenstein, Lisa Beinborn. Relative Importance in Sentence Processing.ACL, 2021

2021

[11] [11]

Towards Faithfully Interpretable NLP Systems.ACL, 2020

Alon Jacovi, Yoav Goldberg. Towards Faithfully Interpretable NLP Systems.ACL, 2020

2020

[12] [12]

LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models.EMNLP, 2023

Huiqiang Jiang et al. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models.EMNLP, 2023

2023

[13] [13]

Earlier Isn’t Always Better

Taehee Jung, Dongyeop Kang, Lucas Mentch, Eduard Hovy. Earlier Isn’t Always Better. EMNLP-IJCNLP, 2019

2019

[14] [14]

Content Selection in Deep Learning Models of Summarization.EMNLP, 2018

Chris Kedzie, Kathleen McKeown, Hal Daum´ e III. Content Selection in Deep Learning Models of Summarization.EMNLP, 2018

2018

[15] [15]

A Trainable Document Summarizer.SIGIR, 1995

Julian Kupiec, Jan Pedersen, Francine Chen. A Trainable Document Summarizer.SIGIR, 1995

1995

[16] [16]

Understanding and Improving Information Preservation in Prompt Compression.Findings of EMNLP, 2025

Weronika Lajewska et al. Understanding and Improving Information Preservation in Prompt Compression.Findings of EMNLP, 2025

2025

[17] [17]

Compressing Context to Enhance Inference Efficiency of Large Language Models.EMNLP, 2023

Yucheng Li et al. Compressing Context to Enhance Inference Efficiency of Large Language Models.EMNLP, 2023

2023

[18] [18]

A Cross-lingual Comparison of Human and Model Relative Word Importance.CLASP, 2022

Felix Morger, Stephanie Brandl, Lisa Beinborn, Nora Hollenstein. A Cross-lingual Comparison of Human and Model Relative Word Importance.CLASP, 2022

2022

[19] [19]

LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression.Findings of ACL, 2024

Zhuoshi Pan et al. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression.Findings of ACL, 2024

2024

[20] [20]

How to Compare Summarizers without Target Length?NeuralGen@NAACL, 2019

Simeng Sun, Ori Shapira, Ido Dagan, Ani Nenkova. How to Compare Summarizers without Target Length?NeuralGen@NAACL, 2019

2019

[21] [21]

Behavioral Analysis of Infor- mation Salience in Large Language Models.Findings of ACL, 2025

Jan Trienes, J¨ org Schl¨ otterer, Junyi Jessy Li, Christin Seifert. Behavioral Analysis of Infor- mation Salience in Large Language Models.Findings of ACL, 2025

2025

[22] [22]

arXiv:2602.00459,Findings of EACL, 2026

What Matters to an LLM? Analysing Information Salience in Large Language Models. arXiv:2602.00459,Findings of EACL, 2026

arXiv 2026

[23] [23]

Searching for Effective Neural Extractive Summarization.ACL, 2019

Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, Xuanjing Huang. Searching for Effective Neural Extractive Summarization.ACL, 2019. 13 Appendix A: Claims withdrawn during this work This work was developed against ten rounds of adversarial internal review. Eleven claims were withdrawn, six of them after we had written them up. We list them because the pa...

2019