Pith. sign in

REVIEW 4 major objections 6 minor 61 references

Principled Content Selection to Generate Diverse and Personalized Multi-Document Summaries

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Explicit, diversity-aware selection of atomic key points improves source coverage in multi-document summarization across four large language models, and the same selection step can be steered by user intent to produce personalized…

desk verdict A clean extraction-DPP-rewrite pipeline that mostly improves coverage on DiverseSumm, but the abstract's 'consistently' claim and the Appendix C.3 significance statement are contradicted by one cell of Table 1. read the letter →

arxiv 2505.21859 v1 pith:QU56PU67 submitted 2025-05-28 cs.CL

Vishakh Padmakumar , Zichao Wang , David Arbour , Jennifer Healey This is my paper
classification cs.CL
keywords multi-documentsummarizationsourcecoveragedeterminantalpointprocessescontentselectionquery-focusedkeyextractionlostinthemiddleLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper contends that the low source coverage of large language models in multi-document summarization is a content-selection failure, not a generation failure. The authors propose a three-step pipeline: break each article into atomic key points with an LLM, select a diverse subset of key points with a determinantal point process (DPP), a statistical model that favors variety among chosen items, and have an LLM rewrite the selected points into a coherent summary. On the DiverseSumm benchmark and a synthetically augmented version, this pipeline raises the fraction of questions answerable from the summary across four LLM families, without increasing summary length. The same selection step can be steered by a user intent, yielding personalized summaries that cover relevant material while keeping overall coverage.

What carries the argument

The load-bearing mechanism is a determinantal point process used for key point selection: given a similarity kernel L built from embeddings of atomic key points, the probability of selecting a subset is proportional to the determinant of its principal submatrix, which penalizes redundant items and therefore favors diversity. LLMs first reduce each document to a list of atomic, context-independent key points, which gives the DPP clean units of selection independent of sentence boundaries. For personalization, the kernel is re-weighted to L' = R L R^T, multiplying each entry by relevance scores of the two key points to the user intent. The paper uses an off-the-shelf DPP inference implementation with exact sampling, so the number of selected key points is determined by the kernel's geometry rather than fixed in advance.

What would settle it

Re-run the evaluation protocol with a judge from a different model family than GPT-4o, or with human annotators, on the same summaries; if LLM+DPP's coverage advantage over LLM-Selected KPs shrinks to statistical insignificance, the paper's central claim would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that explicit, geometry-aware content selection outperforms both end-to-end prompting and LLM-based key point selection. Concretely, after embedding key points with a transformer encoder, the pipeline builds a similarity kernel L and performs MAP inference under a DPP; the selected subset is then rewritten. The authors report consistent coverage gains over the Naive LLM and LLM-Selected KPs baselines on both the original and augmented DiverseSumm datasets. For query-focused summarization, they form the relevance-weighted kernel L' = R L R^T, where R scores each key point against a user intent, and find this improves coverage of intent-relevant questions. A secondary finding is that the gains are not attributable to longer outputs: summary lengths are statistically indistinguishable across methods for three of the four LLMs.

Load-bearing premise

The results depend on the assumption that the GPT-4o judge correctly decides which questions are answerable from a summary and whether the summary's answer matches the source article; this judge is validated on only 100 sampled judgments, and the same model family also generated the test questions.

Editorial extensions

If this is right

  • Content selection can be treated as an explicit optimization step in LLM pipelines, rather than left to the model's attention, mitigating positional biases such as 'lost in the middle'.
  • The number of key points chosen adapts to the embedding geometry, providing automatic compression without a prespecified summary length.
  • Personalization becomes a kernel modification: re-weighting by relevance scores steers the same pipeline toward a user intent, without retraining or changing the generator.
  • The coverage gain holds across four different LLM families, suggesting the benefit comes from the selection mechanism rather than any single model's prompting behavior.
  • Because summary lengths are statistically equal across methods, the improvement reflects better content choice, not a license to produce longer text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism is diversity-by-construction, then other diversity-optimizing selectors, such as maximum marginal relevance or graph-based coreset selection, should show similar coverage gains when substituted for the DPP; the paper does not test this directly.
  • The relevance-weighted result depends on the embedding model used to score key points against the intent; a different relevance scorer could shift the balance between relevance and diversity, so the reported balance is likely specific to the retrieval model chosen.
  • The synthetic question augmentation (ten additional questions per article generated by one LLM) is a reusable outcome: the augmented benchmark could serve future work as a higher-power evaluation set for per-article coverage.
  • Because the evaluation protocol reuses one model family for both question generation and judging, a straightforward external check would replace both with human-annotated questions and an independent judge; the paper does not run this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes a three-step pipeline for multi-document diversity summarization (MDDS): extract atomic key points from each source article with an LLM, select a subset of key points using a determinantal point process (DPP) with a Gaussian kernel over embedding similarities, and rewrite the selected key points into a coherent summary. The authors report that this LLM+DPP pipeline consistently improves source coverage on the DiverseSumm benchmark over a naive prompting baseline and over an LLM-based key-point selection baseline, across GPT-3.5, GPT-4o, Claude-3-Sonnet, and Llama 3.1. They also introduce a query-focused variant in which a relevance vector is multiplied into the DPP kernel, and they report improved coverage of relevant source material. The paper includes ablations of the DPP kernel, a comparison with random key-point sampling, an extractive-sentence DPP baseline, and human validation of the synthetic questions and user intents.

Significance. If the main result holds, the paper demonstrates a simple and training-free way to mitigate lost-in-the-middle bias in long-context multi-document summarization and to steer summaries toward user intents without additional supervision. The design is reproducible from the description: publicly available LLM APIs, the DPPy library, BertScore/DeBERTa embeddings, and the e5 retrieval model are used, and the appendix contains useful negative controls such as random sampling (Appendix C.1) and sentence-level DPP selection (Appendix C.2). The human validation of the augmented questions and generated intents is a genuine attempt to ground the synthetic evaluation. However, the internal inconsistency between Table 1 and the significance claim in Appendix C.3, and the circularity of the relevance labels in the query-focused section, need to be resolved before the claims as stated are fully supported.

major comments (4)
  1. [Table 1, Section 5.1, Appendix C.3] The paper's central claim that LLM+DPP "consistently" improves coverage is contradicted by its own Table 1. On DiverseSumm-Augmented with GPT-3.5, LLM+DPP obtains 0.3845 coverage while the LLM-Selected KPs baseline obtains 0.3849, so the proposed method is worse in that cell. Appendix C.3 nevertheless states that, for DiverseSumm-Augmented, the improvement over LLM-Selected KPs is significant for all four LLMs at p<0.05 using a two-tailed paired t-test. A negative observed mean paired difference cannot yield a significant improvement in the direction claimed. One of the table entry, the significance statement, or the abstract wording must be wrong. Please report per-example paired differences and confidence intervals, and correct the table or the significance claim.
  2. [Section 4.1.3, Section B.2, Section 3.2.3, Table 4] The query-focused evaluation is circular with respect to the relevance model. In Section B.2, the set of relevant questions for each user intent is defined by thresholding scores from intfloat/e5-mistral-7b-instruct at 0.6, and in Section 3.2.3 and Section 4.3 the same embedding model is used to compute the relevance weights in the DPP kernel. Thus Table 4 may primarily demonstrate that relevance-weighted DPP selects content that the e5 model considers relevant, rather than content that is relevant to the user intent in a general sense. The human validation in Section B.2 validates the quality of the generated intents, not the correctness of the relevance labels. An independent human relevance annotation or a second relevance model is needed to support the personalization claim.
  3. [Section 4.2, Appendix A.6] The entire coverage evaluation rests on a GPT-4o LLM-as-judge, and the human validation is limited to 100 sampled judgments with three annotations each. The paper does not report agreement stratified by the LLM that generated the summary or by the summarization method, and the same GPT-4o model was used to generate the augmented questions. A systematic judge-side preference for the multi-step pipeline's output style would be enough to explain part of the reported improvement. Please report judge agreement per model and per method, and consider adding a second judge family or a human-validated subset across all conditions.
  4. [Table 3, Section 5.1] The claim that DPP-based selection improves coverage without increasing summary length is not established for Claude. Table 3 shows that LLM+DPP summaries for Claude average 890.37 words versus 706.50 for LLM-Selected KPs and 601.77 for Naive LLM, and the text states that the length differences are significant for Claude. For one of the four models, the coverage gain could therefore be confounded with summary length. The section title should be qualified, or a length-controlled analysis such as coverage per unit length should be provided.
minor comments (6)
  1. [Table 2, Table 1] Model labels are inconsistent across tables and figures ("GPT 3.5", "GPT-4o", "Claude", "Llama"); use a single consistent notation.
  2. [Section 4.3, Table 2] The Gaussian kernel bandwidth sigma is a free parameter, and the paper does not state how sigma = 1 was chosen or whether the conclusion is stable across a range of values; clarify whether this choice involved the test set.
  3. [Section 3.2.3] The definition L' = R L R^T is immediately followed by an entry-wise repetition of the same formula; remove the redundancy for clarity.
  4. [Appendix B.2] The appendix reports the mean number of retained intents per example but does not state how many stories were dropped entirely because they had fewer than 20 relevant questions; report the final number of examples in DiverseSummRelevance.
  5. [Limitations] The limitation that selection is based purely on diversity applies only to the non-query pipeline, since Section 3.2.3 already incorporates relevance; the wording should be clarified.
  6. [Appendix C.3] The significance tests are reported only as p<0.05; include effect sizes, confidence intervals, and the number of paired examples to allow readers to assess the magnitude of the improvements.

Circularity Check

1 steps flagged · score 6.0 of 10

Query-focused relevance evaluation is built from the same embedding model that drives DPP relevance selection, making Table 4 partly self-confirming; the core MDDS coverage result is independent.

  1. self definitional [Sections 3.2.3, 4.3, Appendix B.2]
    "For each of the intents, we identify the set of relevant questions by scoring the relevance of all DIVERSESUMM-Augmented questions to that particular intent with the trained intfloat/e5-mistral-7b-instruct model. We set the threshold as 0.6 above which a question is deemed relevant. ... we create the relevance vector R, where each entry Ri represents the relevance score of ki ∈ K calculated as frel(ki|quser). ... we score the relevance of different key points to quser using an instruction-tuned retrieval model, intfloat/e5-mistral-7b-instruct"

    In the query-focused setting, the evaluation target Quser is defined by thresholding e5-mistral-7b-instruct relevance scores of questions to the user intent, while LLM+DPP-Relevance builds its selection kernel from the same model's frel scores over keypoints. Both the set of questions that count as 'relevant' and the selection criterion are therefore generated by the same retrieval model; no human annotation validates the relevance filtering, only the intents themselves are validated. The Table 4 improvement of LLM+DPP-Relevance over baselines can thus reflect the method being given direct access to the scoring function used to construct the test set, rather than an independent measurement of user-intent alignment.

full rationale

The paper's primary MDDS contribution, higher source coverage via DPP keypoint selection (Tables 1-3, Figures 2-3), is not circular: coverage is measured against DiverseSumm questions and a GPT-4o judge with 100-sample human validation, while the DPP kernel uses Deberta/BertScore embeddings, an independent signal. The 'consistently improve' wording is undermined by Table 1 (GPT-3.5 on DiverseSumm-Augmented: 0.3845 vs 0.3849), and Appendix C.3's significance claim cannot hold for that cell in the stated direction, but these are internal-consistency/correctness problems, not circularity. The one substantive circular step is the query-focused evaluation: Section B.2 constructs the relevant-question set with intfloat/e5-mistral-7b-instruct, Section 3.2.3 weights DPP selection with the same model's relevance function, and Section 4.3 identifies that model. Consequently, LLM+DPP-Relevance is optimized against the very scoring function that defines the evaluation's positive set, making the personalization results partly self-confirming. The paper's Limitations section appropriately acknowledges LLM-as-judge limitations, but it does not disclose this shared relevance model between method and evaluation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The main non-query result depends on extraction completeness, the diversity-to-coverage assumption, and the LLM judge. The query-focused result additionally depends on e5 as the relevance ground truth and the 0.6 threshold. No new entities are posited; sigma and the relevance threshold are the main free parameters.

free parameters (3)
  • Gaussian kernel bandwidth sigma = 1 (default; 0.1 and 10 in ablation)
    Main results use sigma=1; Table 2 shows coverage varies with sigma, so this hand-selected scale affects the central numbers.
  • Relevance threshold for user-intent questions = 0.6
    Section B.2 uses this threshold on e5 relevance scores to define Quser, the evaluation target for Section 5.2, and to filter intents.
  • Number of synthetic user intents per story = 5 (retained intents average 4.65)
    Section 4.1.3 prompts GPT-4o for 5 intents; the query-focused dataset and Table 4 depend on this synthetic construction.
assumptions (5)
  • domain assumption LLM key point extraction is complete enough to cover all answerable information.
    Section 3.1 depends on a zero-shot prompt to reduce each article to atomic key points; if extraction drops information, no selection strategy can recover it.
  • domain assumption Embedding-space diversity of key points predicts coverage of the benchmark questions.
    Section 3.2.2 selects key points by pairwise similarity in DeBERTa and BertScore space; the paper does not prove this correlates with answer coverage, only evaluates it.
  • domain assumption GPT-4o-as-judge gives valid answerability and correctness labels for all models.
    Section 4.2 relies on GPT-4o binary answerability and correctness judgments; human agreement is checked on only 100 sampled outputs.
  • domain assumption The e5 relevance model provides a valid operationalization of user-intent relevance.
    Section 3.2.3 and Section B.2 both use intfloat/e5-mistral-7b-instruct to define and optimize relevance; this assumption is load-bearing for Section 5.2.
  • standard math DPPy spectral sampling correctly samples from the DPP defined by kernel L.
    Section 4.3 uses DPPy default exact sampling; no implementation verification beyond the library citation is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Principled Content Selection to Generate Diverse and Personalized Multi-Document Summaries." pith.science (2026). https://pith.science/paper/QU56PU67

@misc{pith2026250521859,
  author       = {Pith},
  title        = {Pith review of: Principled Content Selection to Generate Diverse and Personalized Multi-Document Summaries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QU56PU67}},
  note         = {Machine review of arXiv:2505.21859}
}
read the original abstract

While large language models (LLMs) are increasingly capable of handling longer contexts, recent work has demonstrated that they exhibit the "lost in the middle" phenomenon (Liu et al., 2024) of unevenly attending to different parts of the provided context. This hinders their ability to cover diverse source material in multi-document summarization, as noted in the DiverseSumm benchmark (Huang et al., 2024). In this work, we contend that principled content selection is a simple way to increase source coverage on this task. As opposed to prompting an LLM to perform the summarization in a single step, we explicitly divide the task into three steps -- (1) reducing document collections to atomic key points, (2) using determinantal point processes (DPP) to perform select key points that prioritize diverse content, and (3) rewriting to the final summary. By combining prompting steps, for extraction and rewriting, with principled techniques, for content selection, we consistently improve source coverage on the DiverseSumm benchmark across various LLMs. Finally, we also show that by incorporating relevance to a provided user intent into the DPP kernel, we can generate personalized summaries that cover relevant source information while retaining coverage.

Figures

Figures reproduced from arXiv: 2505.21859 by the authors.

Figure 1
Figure 1. Overview of the MDDS task (Section 2), which aims to generate a summary from a set of source documents with an optional user intent. Compared to (a) prompting an LLM to perform MDDS in a single step (Naive LLM) and other baselines, (b) our method (LLM + DPP) first extracts atomic key points from each document, then explicitly selects content using DPPs to ensure diversity and relevance before rewriting them into a s… view at source ↗
Figure 2
Figure 2. Studying the ’lost-in-the-middle’ phenomenon by plotting coverage of different source articles by index [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Distribution of source documents covered by [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Case study of LLM + DPP (Section 5.2) selecting key points that are diverse and yet relevant to two different user intents (Section 3.2.3) and evaluation of the summaries via question-answering (Section 4.2) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: TSNE visualization of the key points selected for the two user intents in Figure [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparing using a k-DPP with uniform random sampling for key point selection for DIVERS￾ESUMM, varying k, across 4 different LLMs (Section C). The k-DPP consistently outperforms uniform random sampling, showing the value in sampling while consid￾ering the learned repre…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 34 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  4. [4]

    Anthropic . 2024 a . https://www.anthropic.com/news/claude-3-5-sonnet Claude 3.5 Sonnet . Technical report, Anthropic. Accessed: 2024-06-23

    work page 2024

  5. [5]

    Anthropic . 2024 b . https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf The Claude 3 Model Family: Opus, Sonnet, Haiku . Technical report, Anthropic. Accessed: 2024-05-23

    work page 2024

  6. [6]

    Nishant Balepur, Feng Gu, Abhilasha Ravichander, Shi Feng, Jordan Boyd-Graber, and Rachel Rudinger. 2024 a . Reverse question answering: Can an llm write a question so hard (or bad) that it can't answer? arXiv preprint arXiv:2410.15512

    work page Pith review arXiv 2024

  7. [7]

    Nishant Balepur, Vishakh Padmakumar, Fumeng Yang, Shi Feng, Rachel Rudinger, and Jordan Boyd-Graber. 2024 b . Whose boat does it float? improving personalization in preference tuning via inferred user personas

    work page 2024

  8. [8]

    Regina Barzilay, Kathleen McKeown, and Michael Elhadad. 1999. Information fusion in the context of multi-document summarization. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics, pages 550--557

    work page 1999

Show all 61 references
  1. [9]

    Adithya Bhaskar, Alex Fabbri, and Greg Durrett. 2023. Prompted opinion summarization with gpt-3.5. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9282--9300

    2023

  2. [10]

    Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. https://openreview.net/forum?id=7Ttk3RzDeu Booookscore: A systematic exploration of book-length summarization in the era of LLM s . In The Twelfth International Conference on Learning Representations

    2024

  3. [11]

    Laming Chen, Guoxin Zhang, and Eric Zhou. 2018. Fast greedy map inference for determinantal point process to improve recommendation diversity. Advances in Neural Information Processing Systems, 31

    2018

  4. [12]

    Jianpeng Cheng and Mirella Lapata. 2016. https://doi.org/10.18653/v1/P16-1046 Neural summarization by extracting sentences and words . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 484--494, Berlin, Ge...

    2016 doi

  5. [13]

    Sangwoo Cho, Logan Lebanoff, Hassan Foroosh, and Fei Liu. 2019 a . https://doi.org/10.18653/v1/P19-1098 Improving the similarity measure of determinantal point processes for extractive multi-document summarization . In Proceedings of the 57th Annual Meeting of the Association ...

    2019 doi

  6. [14]

    Sangwoo Cho, Chen Li, Dong Yu, Hassan Foroosh, and Fei Liu. 2019 b . https://doi.org/10.18653/v1/D19-5412 Multi-document summarization with determinantal point processes and contextualized representations . In Proceedings of the 2nd Workshop on New Frontiers in Summarization, ...

    2019 doi

  7. [15]

    Hoa Trang Dang. 2005. Overview of duc 2005. In Proceedings of the document understanding conference, volume 2005, pages 1--12. Citeseer

    2005

  8. [16]

    Hal Daum \'e III and Daniel Marcu. 2006. Bayesian query-focused summarization. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 305--312

    2006

  9. [17]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  10. [18]

    Alexander Richard Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pa...

    2019

  11. [19]

    Lu Gan, Diana Nurbakova, L\' e a Laporte, and Sylvie Calabretto. 2020. https://doi.org/10.1145/3397271.3401213 Enhancing recommendation diversity using determinantal point processes on knowledge graphs . SIGIR '20, page 2001–2004, New York, NY, USA. Association for Computing Machinery

    2020

  12. [20]

    Guillaume Gautier, Guillermo Polito, R \'e mi Bardenet, and Michal Valko. 2019. Dppy: Dpp sampling with python. Journal of Machine Learning Research, 20(180):1--7

    2019

  13. [21]

    Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209.12356

    2022 arXiv

  14. [22]

    Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. In The Eleventh International Conference on Learning Representations

  15. [23]

    Kai Hong and Ani Nenkova. 2014. Improving the estimation of word importance for news multi-document summarization. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 712--721

    2014

  16. [24]

    Kung-Hsiang Huang, Philippe Laban, Alexander Richard Fabbri, Prafulla Kumar Choubey, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. 2024. Embrace divergence for richer insights: A multi-document summarization benchmark and a case study on summarizing diverse information from ...

    2024

  17. [25]

    Gregory Kamradt. 2023. https://github.com/gkamradt/LLMTest_NeedleInAHaystack/tree/main Needle In A Haystack - pressure testing LLM s . Github

    2023

  18. [26]

    Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. One thousand and one pairs: A" novel" challenge for long-context language models. arXiv preprint arXiv:2406.16264

    2024 arXiv

  19. [27]

    Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. Fables: Evaluating faithfulness and content selection in book-length summarization. In Conference on Language Modeling

    2024

  20. [28]

    Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. Longeval: Guidelines for human evaluation of faithfulness in long-form summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for ...

    2023

  21. [29]

    Alex Kulesza, Ben Taskar, et al. 2012. Determinantal point processes for machine learning. Foundations and Trends in Machine Learning , 5(2--3):123--286

    2012

  22. [30]

    Zongxia Li, Ishani Mondal, Huy Nghiem, Yijun Liang, and Jordan Lee Boyd-Graber. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.548 PEDANTS : Cheap but effective and interpretable answer equivalence . In Findings of the Association for Computational Linguistics: EMNLP 20...

    2024 doi

  23. [31]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157--173

    2024

  24. [32]

    Yao Lu, Yue Dong, and Laurent Charlin. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.648 Multi- XS cience: A large-scale dataset for extreme multi-document summarization of scientific articles . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language...

    2020 doi

  25. [33]

    Kathleen McKeown and Dragomir R Radev. 1995. Generating summaries of multiple news articles. In Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, pages 74--82

    1995

  26. [34]

    OpenAI . 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  27. [35]

    Paul Over and James Yen. 2004. An introduction to duc-2004. National Institute of Standards and Technology

    2004

  28. [36]

    Karolina Owczarzak and Hoa Trang Dang. 2011. Overview of the tac 2011 summarization track: Guided task and aesop task. In Proceedings of the Text Analysis Conference (TAC 2011), Gaithersburg, Maryland, USA, November

    2011

  29. [37]

    Does writing with language models reduce content diversity? In The Twelfth International Conference on Learning Representations

    Vishakh Padmakumar and He He. Does writing with language models reduce content diversity? In The Twelfth International Conference on Learning Representations

  30. [38]

    Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, et al. 2022. Quality: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American ...

    2022

  31. [39]

    Laura Perez-Beltrachini and Mirella Lapata. 2021. Multi-document summarization with determinantal point process attention. Journal of Artificial Intelligence Research, 71:371--399

    2021

  32. [40]

    Dragomir R Radev, Hongyan Jing, Ma gorzata Sty \'s , and Daniel Tam. 2004. Centroid-based summarization of multiple documents. Information Processing & Management, 40(6):919--938

    2004

  33. [41]

    Dragomir R Radev and Kathleen R McKeown. 1998. Generating natural language summaries from multiple on-line sources. Computational Linguistics, 24(3):470--500

    1998

  34. [42]

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arX...

    2024 arXiv

  35. [43]

    Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. 2022. Scrolls: Standardized comparison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language...

    2022

  36. [44]

    Yun-Zhu Song, Yi-Syuan Chen, and Hong-Han Shuai. 2022. https://doi.org/10.18653/v1/2022.naacl-main.120 Improving multi-document summarization through referenced flexible extraction with credit-awareness . In Proceedings of the 2022 Conference of the North American Chapter of t...

    2022 doi

  37. [45]

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021. https://openreview.net/forum?id=qVyeW-grC2k Long range arena : A benchmark for efficient transformers . In International Conference o...

    2021

  38. [46]

    Huy Quoc To, Hung-Nghiep Tran, Andr'e Greiner-Petter, Felix Beierle, and Akiko Aizawa. 2024. Skt5scisumm-a hybrid generative approach for multi-document scientific summarization. arXiv preprint arXiv:2402.17311

    2024 arXiv

  39. [47]

    Jesse Vig, Alexander Richard Fabbri, Wojciech Kry \'s ci \'n ski, Chien-Sheng Wu, and Wenhao Liu. 2022. Exploring neural models for query-focused summarization. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1455--1468

    2022

  40. [48]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368

    2023 arXiv

  41. [49]

    Peng Wang, Xiaobin Wang, Chao Lou, Shengyu Mao, Pengjun Xie, and Yong Jiang. 2024. Effective demonstration annotation for in-context learning via language model-based determinantal point process. arXiv preprint arXiv:2408.02103

    2024 arXiv

  42. [50]

    Chi, and Jennifer Gillenwater

    Mark Wilhelm, Ajith Ramanathan, Alexander Bonomo, Sagar Jain, Ed H. Chi, and Jennifer Gillenwater. 2018. https://doi.org/10.1145/3269206.3272018 Practical diversified recommendations on youtube with determinantal point processes . In Proceedings of the 27th ACM International C...

    2018

  43. [51]

    Zhao Yang, Yuanzhe Zhang, Dianbo Sui, Cao Liu, Jun Zhao, and Kang Liu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.331 Representative demonstration selection for in-context learning with two-stage determinantal point process . In Proceedings of the 2023 Conference on Emp...

    2023 doi

  44. [52]

    Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. https://proceedings.mlr.press/v202/ye23c.html Compositional exemplars for in-context learning . In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machin...

    2023

  45. [53]

    Jianmin Zhang, Jiwei Tan, and Xiaojun Wan. 2018. Towards a neural network approach to abstractive multi-document summarization. arXiv preprint arXiv:1804.09010

    2018 arXiv

  46. [54]

    Shiyue Zhang, David Wan, and Mohit Bansal. 2023. Extractive is not faithful: An investigation of broad unfaithfulness problems in extractive summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages...

    2023

  47. [55]

    Weinberger, and Yoav Artzi

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  48. [56]

    Xu Zhang, Dinghao Jing, and Xiaojun Wan. 2024. https://arxiv.org/abs/2403.00292 Enhancing jailbreak attacks with diversity guidance . Preprint, arXiv:2403.00292

    2024 arXiv

  49. [57]

    Gonzalez, Ion Stoica, and Hao Zhang

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. 2024. https://openreview.net/forum?id=BOfDKxfwt0 LMSYS -chat-1m: A large-scale real-world LLM convers...

    2024

  50. [58]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  51. [59]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  52. [60]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  53. [61]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.