Pith. sign in

REVIEW 5 major objections 6 minor 59 references

LLMs as Architects and Critics for Multi-Source Opinion Summarization

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multi-source opinion summaries, generated by LLMs from product metadata plus reviews, are preferred by 87% of users, and a structured prompt reaches $\rho = 0.74$ agreement with human evaluators.

desk verdict M-OS-EVAL is a solid benchmark; the 87% user preference claim is unsupported as reported. read the letter →

arxiv 2507.04751 v1 pith:H5Y23KMH submitted 2025-07-07 cs.CL

Anuj Attri , Arnav Attri , Pushpak Bhattacharyya , Suman Banerjee , Amey Patil , Muthusamy Chelliah , Nikesh Garera This is my paper
classification cs.CL
keywords multi-sourceopinionsummarizationLLM-basedevaluationpromptengineeringreference-freebenchmarke-commerceproductmetadatauserstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that opinion summarization in e-commerce should go beyond reviews by folding in product descriptions, key features, specifications, and average ratings, producing what the authors call a multi-source opinion summary (M-OS). To support this, it introduces a proprietary product-metadata dataset of roughly 25,000 products and an evaluation benchmark of 4,900 human ratings across seven dimensions: fluency, coherence, relevance, faithfulness, aspect coverage, sentiment consistency, and specificity. The paper's central claims are that large language models can both generate such summaries and evaluate them without reference summaries when guided by structured prompts, and that users prefer the result: on average 87% of 300 study participants chose M-OS over a review-only summary. The best evaluator prompt reaches an average Spearman correlation of $\rho = 0.74$ with human judgment, which the authors report as surpassing previous prompt-based evaluation methods.

What carries the argument

The central mechanism is the M-OS-PROMPTS family. A generation prompt (M-OS-GEN-PROMPT) instructs the LLM to balance objective product data (title, description, key features, specifications, ratings) with subjective customer reviews, while the evaluation prompts (M-OS-EVAL-PROMPTS) share a four-part architecture: a system message, a task description, evaluation criteria, and an evaluation step. Two variants are introduced: OMNI-PROMPT, a single modular template whose Metric component can be swapped to assess any dimension, and SPECTRA-PROMPTS, seven dimension-specific prompts. Scores come from a weighted scoring function that estimates each candidate score's probability from roughly 100 samples per summary, effectively turning discrete 1–5 scores into a mean. The paper argues that the structured, step-by-step format—explicit percentage ranges for each score level and a demand for justification—reduces the score inflation observed in prior baselines and is what carries the $\rho = 0.74$ average correlation.

What would settle it

Re-run the user study with comparison summaries produced by the paper's strongest generator prompted on reviews alone; if the preference margin narrows to near chance, the 87% figure reflects a weak baseline rather than the value of multi-source information.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes M-OS as a new task definition: a summary must cover both subjective opinions from reviews and objective product attributes from metadata, and it should serve purchasing decisions. It reports that the strongest generator among 14 benchmarked LLMs is Qwen2.5-72B-Instruct, with an average annotator rating of 4.186 across the seven dimensions, edging out GPT-4o at 4.169. For evaluation, the OMNI-PROMPT paired with GPT-4o achieves the highest average Spearman correlation ($\rho = 0.74$) with human ratings, outperforming both the metric-dependent SPECTRA-PROMPTS and the earlier prompt baselines. The user study finds 86.6% overall preference for M-OS across 6,000 judgments, with per-criterion preference between 85.7% and 88.4%, and the chi-square statistic $\chi^2 = 3126.83$ ($df = 1$, $p < .001$) rejects equal preference.

Load-bearing premise

The user-preference result relies on the 'traditional opinion-summary method' used as the comparison being a fair, representative baseline, but the paper never identifies which model or prompt produced those summaries.

Editorial extensions

If this is right

  • E-commerce platforms could replace separate metadata pages and review sections with a single generated summary per product, since the user study indicates M-OS reduces the need for additional lookups.
  • Reference-free LLM evaluation at $\rho = 0.74$ could substitute for expensive human annotation when comparing future multi-source summarization models along the same seven dimensions.
  • Open-source models are viable for both generation and evaluation: Qwen2.5-72B-Instruct outperformed GPT-4o in generation, and Llama-3.1-70B-Instruct and Mistral-7B-Instruct-v0.2 approached GPT-4o in evaluation, which matters for deployments that cannot use proprietary APIs.
  • Dimension-specific prompts (SPECTRA-PROMPTS) and the modular prompt (OMNI-PROMPT) each beat their respective baselines, indicating that structured evaluation instructions with explicit score ranges reduce score inflation.
  • The M-OS-EVAL benchmark, with 4,900 human ratings across seven dimensions, provides a reusable testbed for future multi-source opinion summarization systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural follow-up would be to ablate the metadata sources (specifications, ratings, descriptions) to test which one drives user preference; the paper reports overall preferences but no per-source contribution analysis.
  • The $\rho = 0.74$ correlation could degrade on out-of-distribution product categories or non-English reviews; a stress test along those lines would reveal whether the prompt framework generalizes beyond the curated e-commerce data.
  • The seven evaluation dimensions could transfer to other multi-document summarization tasks, such as legal or medical briefs, where faithfulness and aspect coverage carry similar weight, though the criteria would need domain-specific rewriting.
  • Because the evaluation benchmark draws on only 50 products, scaling it to several hundred would show whether the OMNI-PROMPT advantage is stable or concentrated in particular product categories.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Multi-Source Opinion Summarization (M-OS), in which LLMs generate summaries that integrate customer reviews with product metadata such as descriptions, key features, specifications, and ratings. It introduces M-OS-DATA, a proprietary product-metadata dataset; M-OS-EVAL, an evaluation benchmark with 7 dimensions and two rounds of expert annotation; and two evaluation prompt frameworks, OMNI-PROMPT and SPECTRA-PROMPTS. The authors benchmark 14 LLMs as summary generators and 5 LLMs as automatic evaluators, reporting an average Spearman correlation of 0.74 for OMNI-GPT-4o against human judgment, and a user study in which 87% of participants preferred M-OS summaries over traditional opinion summaries. The central claims are that M-OS improves user engagement and that the proposed prompts outperform previous evaluation methodologies.

Significance. If the results are supported, the paper would make a useful contribution: it provides a new task formulation, a benchmark with expert annotation, a large product-metadata resource, and evidence that reference-free LLM evaluation can align with human judgments for multi-source summaries. A particular strength is that the headline correlation is measured against human annotations rather than against model-generated gold labels, and the annotation procedure includes a second adjudication round that raises inter-rater agreement from 0.70 to 0.86. The generation prompts were not fitted to the preference outcome, which reduces one common circularity concern. However, the user-engagement claim and the claim of superiority over previous methodologies are not yet established at the level of evidence the paper asserts, for the reasons detailed below.

major comments (5)
  1. [Section 7 and Appendix C] The user-study comparator is never specified. Section 7 states only that participants compared 'M-OS vs. traditional opinion-summary method', and Appendix C gives no model, prompt, or generation settings for the traditional summaries, although Appendix E.1 gives full implementation details for M-OS generation. Without a matched baseline, the 87% preference figure cannot be attributed to M-OS as a method: if the comparator was produced by a weaker model or a prompt that intentionally omitted metadata, the result measures information asymmetry rather than the value of M-OS. The outcome instrument compounds this problem, because all five questions in Appendix C D.2 ask about product specifications, technical details, and reduced need to look up additional product information, which are exactly the features M-OS adds by construction. The paper should identify the comparator, provide its generation configuration, and either use evaluation questions that do not presuppose the value of metadata or analyze the metadata-specific and non-metadata-specific questions separately.
  2. [Appendix C D.3] The chi-square test treats all 6,000 preference judgments as independent, but the judgments are nested in 300 participants, each contributing 20 judgments across 4 product pairs and 5 questions. Judgments from the same participant are likely correlated, so the reported chi-square value of 3126.83 with df=1 and p<.001 substantially overstates the statistical significance, and the derived Cramer's V=0.72 inherits the same problem. The authors should report a participant-level analysis, such as a mixed-effects logistic regression with random intercepts for participants and products, or per-participant preference rates with confidence intervals. This is load-bearing because the 87% engagement claim in the abstract rests on this significance test.
  3. [Section 6.2, Table 6, Eq. (2)] The headline average Spearman correlation of 0.74 is computed as an average of per-product correlations, each over only 14 summaries, and no confidence intervals or measures of dispersion are reported. The improvement over the OP-I-PROMPT baseline with GPT-4o (0.70 in Table 6) is small, and the asterisks in Table 6 mark only individual dimensions, not the average across dimensions. The claim in the abstract that M-OS-PROMPTS 'surpass the performance of previous methodologies' is therefore not statistically supported. The authors should report bootstrap or across-product confidence intervals for the average correlation, and a significance test for the difference between OMNI-PROMPT and the baselines, ideally accounting for the non-independence of the 14 summaries per product.
  4. [Section 3.3 and Appendix E.2] Equation (1) motivates a scoring function by sampling n outputs (n≈100) per input to estimate the score distribution p(s_k), but Appendix E.2 states that a temperature of 0.0 was used for evaluation. At temperature 0.0, repeated sampling is deterministic, so the 100 samples would be identical and p(s_k) would be degenerate, making the sampling-based justification for Eq. (1) inapplicable. The authors should clarify whether temperature was actually nonzero during the 100 evaluations, or revise the description of the scoring procedure to match the implementation. This affects the validity of all LLM-evaluator scores that feed into Tables 4 and 6.
  5. [Section 6.2 and Related Work] The comparison baselines OP-PROMPTS and OP-I-PROMPT come from Siledar et al. (2024a), which shares several co-authors with this submission. Since the abstract claims to surpass 'previous methodologies' and the paper calls these baselines 'state-of-the-art', the comparison is at present only against the authors' own prior prompting variants. The authors should either add an independent baseline from another group, or temper the claim to 'surpass the previously proposed OP-PROMPTS and OP-I-PROMPT baselines'.
minor comments (6)
  1. [Abstract and Section 4.2] The abstract reports '4,900 summary annotations' for M-OS-EVAL, while Section 4.2 states 14,700 total ratings (3 raters × 50 products × 14 summaries × 7 dimensions). Please clarify whether 4,900 refers to summary-level scores after averaging raters, and state this consistently.
  2. [Appendix D] The list of models includes 'close-sourced' (typo for 'closed-source'), says the API model is GPT-4, while the experiments use GPT-4o, and lists Mistral-Nemo-Instruct-2407 and Qwen2.5-14B-Instruct, which do not appear in Table 5. Please reconcile the model list with the models actually reported.
  3. [Section 3.2] The text contains duplicated phrases: 'specificityand specificity' and 'assessment guidelines guidelines'. These should be corrected.
  4. [Appendix A] In the definition of sentiment consistency, 'the M-OSt common sentiment' appears to be a typo for 'the most common sentiment'.
  5. [Appendix F] Figure 4 is referenced as 'Figure ??' in the text; the cross-reference should be fixed.
  6. [Tables 4 and 6] The tables report Spearman and Kendall correlations but give no confidence intervals or measure of variability across products, and the footnote says '*' indicates p<0.05 without describing the significance test. At least one sentence should state how significance was computed.

Circularity Check

1 steps flagged · score 6.0 of 10

The 87% user-preference claim is largely definitional because the survey questions ask exactly for the metadata M-OS adds; the ρ=0.74 correlation claim is independently grounded in human annotations.

  1. self definitional [Abstract; §7 User Study: M-OS Effectiveness; Appendix C, D.2 Evaluation Questions]
    "M-OS extends beyond traditional opinion summarization by incorporating additional sources of product metadata such as descriptions, key features, specifications, and ratings, alongside reviews. … Information Comprehensiveness: “Which summary type (M-OS or Opinion Summary) provides a more complete understanding of both product specifications and customer experiences?”"

    M-OS is defined precisely by adding specifications, features, and ratings to reviews, while the comparison arm is called 'traditional opinion-summary method' and is never concretely specified. Every preference question asks about the exact added content: 'more complete understanding of … product specifications', 'understand both technical specifications and real-world performance', and 'reduce your need to look up additional product information'. With a competent generation prompt, an M-OS summary must contain some specification-level facts that a review-only opinion summary cannot, so the reported 87% preference is largely a restatement of the input design rather than an independent test of engagement.

full rationale

The correlation headline (average Spearman ρ = 0.74) is not circular: the ground truth is human annotation of summaries, which is external to the LLM evaluators, and the prompts were not fitted to those annotations. The M-OS-EVAL benchmark is self-created, but the claimed alignment is measured against rater judgments, not against the evaluators' own outputs. The choice of OP-PROMPTS and OP-I-PROMPT as 'current state-of-the-art' baselines comes from Siledar et al. (2024a), which has overlapping authorship; this is a mild self-citation and benchmark-selection concern, but not a derivation from that citation, and it does not affect the external human-judgment grounding of the correlation result. The separate engagement claim, however, is partially circular: M-OS is defined by adding metadata, and the user-study questions ask users to choose the summary that provides more specification understanding and reduces the need to look up product information—exactly the attributes M-OS adds by construction, with the 'traditional opinion-summary method' left unspecified. The 87% figure is therefore not an independent discovery about user engagement; it is largely an artifact of the outcome instrument. Other completeness defects (mislabeled Appendix C subsections, undefined 'Figure ??' in Appendix F) are not circularity. Overall: one headline claim reduces by construction while the other is independently measured, giving score 6.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on four domain assumptions: expert human ratings are a valid gold standard; the seven chosen dimensions cover summary quality; the G-Eval sampling distribution is meaningful; and the unnamed user-study baseline is representative. The main free parameters are hand-set evaluation settings (n=100 samples, score percentage ranges, decoding hyperparameters). No new physical or formal entities are introduced; the dataset and prompts are artifacts rather than postulated constructs.

free parameters (3)
  • Number of LLM evaluation samples (n) = 100
    Section 3.3 and Appendix E.2 set n=100 sampled evaluations per summary to estimate the score distribution; the choice is hand-set and affects the stability of the LLM scores that enter every correlation.
  • Score-level percentage ranges in evaluation prompts = Not disclosed numerically
    Appendix B.2 says 'precise percentage ranges for each score level' are built into the prompts to prevent score inflation; these anchors are hand-designed and directly shape all LLM evaluator scores, but the actual ranges are not printed.
  • Decoding hyperparameters = temperature=0.2 (generation), temperature=0.0 (evaluation), top_k=25, top_p=0.95
    Appendix E.1 states these were selected 'after extensive experimentation', i.e., tuned on the task; they influence generation quality and thus the human-rated and user-studied summaries.
assumptions (5)
  • domain assumption Human ratings from three in-house expert raters, adjudicated to Krippendorff alpha 0.86, are a valid gold standard for summary quality.
    All Spearman correlations treat these ratings as ground truth; the raters are three male students aged 24-32 from the same lab, and no external validation of the gold standard is given (Sections 4.2-4.3).
  • domain assumption The seven evaluation dimensions fully capture what makes a multi-source opinion summary good.
    M-OS-EVAL is built on these seven dimensions defined in Appendix A, with dimension choices taken from prior summarization evaluation work; no dimensionality or coverage analysis is provided.
  • ad hoc to paper Averaging n=100 LLM score samples gives a valid estimate of the score distribution in Equation 1.
    Section 3.3 adopts the G-Eval sampling approach without validating that the sampled probabilities p(s_k) are well-calibrated for these prompts and this data domain.
  • domain assumption The 'traditional opinion-summary method' used in the user study is a representative baseline for comparison.
    Section 7 and Appendix C never name the model or prompt that produced the comparison summaries; the 87% preference claim assumes this baseline is fair.
  • domain assumption Per-product summary-level correlation in Equation 2 is an appropriate way to aggregate LLM versus human agreement.
    Section 3.4 averages correlations computed over the 14 summaries within each product, which yields noisy per-product Spearman values; the paper does not justify this aggregation over the alternative of pooling all 700 summaries.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMs as Architects and Critics for Multi-Source Opinion Summarization." pith.science (2026). https://pith.science/paper/H5Y23KMH

@misc{pith2026250704751,
  author       = {Pith},
  title        = {Pith review of: LLMs as Architects and Critics for Multi-Source Opinion Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H5Y23KMH}},
  note         = {Machine review of arXiv:2507.04751}
}
read the original abstract

Multi-source Opinion Summarization (M-OS) extends beyond traditional opinion summarization by incorporating additional sources of product metadata such as descriptions, key features, specifications, and ratings, alongside reviews. This integration results in comprehensive summaries that capture both subjective opinions and objective product attributes essential for informed decision-making. While Large Language Models (LLMs) have shown significant success in various Natural Language Processing (NLP) tasks, their potential in M-OS remains largely unexplored. Additionally, the lack of evaluation datasets for this task has impeded further advancements. To bridge this gap, we introduce M-OS-EVAL, a benchmark dataset for evaluating multi-source opinion summaries across 7 key dimensions: fluency, coherence, relevance, faithfulness, aspect coverage, sentiment consistency, specificity. Our results demonstrate that M-OS significantly enhances user engagement, as evidenced by a user study in which, on average, 87% of participants preferred M-OS over opinion summaries. Our experiments demonstrate that factually enriched summaries enhance user engagement. Notably, M-OS-PROMPTS exhibit stronger alignment with human judgment, achieving an average Spearman correlation of \r{ho} = 0.74, which surpasses the performance of previous methodologies.

Figures

Figures reproduced from arXiv: 2507.04751 by the authors.

Figure 1
Figure 1. Pipeline of our experiments involving generation, evaluation, and correlation analysis. LLMgen (LLM used as a generator), which generates M-OS using product metadata, (comprising product title, description, key features, specifications, average ratings, and customer reviews) guided by the Summary Generation Instructions (M-OS-GEN-PROMPT). LLMeval (LLM used as an evaluator), which evaluates the summaries based on Sum… view at source ↗
Figure 2
Figure 2. Preference analysis comparing Multi-Source Opinion Summaries (M-OS) versus traditional opinion summaries across product categories (N = 300). Bars represent the mean preference percentage across five evaluation questions per category. Statistical signifi￾cance: χ 2 = 3126.83 (df = 1, p < .001). • Purchase Decision Support: “Which sum￾mary format provides a better balance of tech￾nical details and user experiences to… view at source ↗
Figure 3
Figure 3. Structure of Dependent Prompts [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Structure of Inependent Prompts [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 24 canonical work pages

  1. [1]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  2. [2]

    Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.528 Aspect-controllable opinion summarization . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6578--6593, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics

    work page doi:10.18653/v1/2021.emnlp-main.528 2021

  3. [3]

    Reinald Kim Amplayo and Mirella Lapata. 2020 a . Unsupervised opinion summarization with noising and denoising. arXiv preprint arXiv:2004.10150

    work page Pith review arXiv 2020

  4. [4]

    Reinald Kim Amplayo and Mirella Lapata. 2020 b . https://doi.org/10.18653/v1/2020.acl-main.175 Unsupervised opinion summarization with noising and denoising . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1934--1945, Online. Association for Computational Linguistics

    work page doi:10.18653/v1/2020.acl-main.175 2020

  5. [5]

    Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. 2021. https://doi.org/10.1162/tacl_a_00366 Extractive opinion summarization in quantized transformer spaces . Transactions of the Association for Computational Linguistics, 9:277--293

    work page doi:10.1162/tacl_a_00366 2021

  6. [6]

    Anthropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Introducing claude 3.5 sonnet . Accessed: 2024-12-16

    work page 2024

  7. [7]

    Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.751 Re-evaluating evaluation in text summarization . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347--9359, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.emnlp-main.751 2020

  8. [8]

    Adithya Bhaskar, Alex Fabbri, and Greg Durrett. 2023. https://doi.org/10.18653/v1/2023.findings-acl.591 Prompted opinion summarization with GPT -3.5 . In Findings of the Association for Computational Linguistics: ACL 2023, pages 9282--9300, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-acl.591 2023

Show all 59 references
  1. [9]

    Arthur Bra z inskas, Mirella Lapata, and Ivan Titov. 2020 a . Few-shot learning for opinion summarization. arXiv preprint arXiv:2004.14884

    2020 arXiv

  2. [11]

    Arthur Bra z inskas, Mirella Lapata, and Ivan Titov. 2020 c . https://doi.org/10.18653/v1/2020.acl-main.461 Unsupervised opinion summarization as copycat-review generation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5151-...

    2020 doi

  3. [13]

    Cheng-Han Chiang and Hung-yi Lee. 2023 b . https://doi.org/10.18653/v1/2023.acl-long.870 Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2023 doi

  4. [14]

    Cheng-Han Chiang and Hung-yi Lee. 2023 c . https://doi.org/10.18653/v1/2023.findings-emnlp.599 A closer look into using large language models for automatic evaluation . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8928--8942, Singapore. Assoc...

    2023 doi

  5. [15]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90\

    2023

  6. [16]

    Eric Chu and Peter Liu. 2019 a . Meansum: a neural model for unsupervised multi-document abstractive summarization. In International conference on machine learning, pages 1223--1232. PMLR

    2019

  7. [17]

    Eric Chu and Peter J. Liu. 2019 b . http://arxiv.org/abs/1810.05739 Meansum: A neural model for unsupervised multi-document abstractive summarization

    2019 arXiv

  8. [18]

    Clark, Chris Dyer, Alon Lavie, and Noah A

    Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. https://aclanthology.org/P11-2031 Better hypothesis testing for statistical machine translation: Controlling for optimizer instability . In Proceedings of the 49th Annual Meeting of the Association for Computa...

    2011

  9. [19]

    Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences, 2nd edition. Lawrence Erlbaum Associates, Hillsdale, NJ

    1988

  10. [20]

    Hady Elsahar, Maximin Coavoux, Jos Rozen, and Matthias Gall \'e . 2021. https://doi.org/10.18653/v1/2021.eacl-main.141 Self-supervised and controlled multi-document opinion summarization . In Proceedings of the 16th Conference of the European Chapter of the Association for Com...

    2021 doi

  11. [21]

    G \"u nes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research, 22:457--479

    2004

  12. [22]

    Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev

    Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. http://arxiv.org/abs/2007.12626 Summeval: Re-evaluating summarization evaluation

    2021 arXiv

  13. [24]

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023 b . http://arxiv.org/abs/2302.04166 Gptscore: Evaluate as you desire

    2023 arXiv

  14. [25]

    Dan Gillick and Yang Liu. 2010. https://aclanthology.org/W10-0722 Non-expert evaluation of summarization systems is risky . In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with A mazon ' s Mechanical Turk , pages 148--151, Los Angeles. Associ...

    2010

  15. [26]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  16. [27]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  17. [28]

    Han Jiang, Rui Wang, Zhihua Wei, Yu Li, and Xinpeng Wang. 2023 b . https://doi.org/10.18653/v1/2023.findings-emnlp.375 Large-scale and multi-perspective opinion summarization with diverse review subsets . In Findings of the Association for Computational Linguistics: EMNLP 2023...

    2023 doi

  18. [29]

    Hyun Duk Kim, Kavita Ganesan, Parikshit Sondhi, and ChengXiang Zhai. 2011. Comprehensive review of opinion summarization

    2011

  19. [30]

    Tom Kocmi and Christian Federmann. 2023. https://aclanthology.org/2023.eamt-1.19 Large language models are state-of-the-art evaluators of translation quality . In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193--203, Tam...

    2023

  20. [31]

    Klaus Krippendorff. 2011. https://api.semanticscholar.org/CorpusID:59901023 Computing krippendorff's alpha-reliability

    2011

  21. [32]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. http://arxiv.org/abs/1910.13461 Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

    2019 arXiv

  22. [33]

    Haoran Li, Peng Yuan, Song Xu, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2020. https://doi.org/10.1609/aaai.v34i05.6332 Aspect-aware multimodal summarization for chinese e-commerce products . Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8188--8195

    2020 doi

  23. [34]

    Chin-Yew Lin. 2004 a . https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  24. [35]

    Chin-Yew Lin. 2004 b . https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  25. [36]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 a . https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Langua...

    2023 doi

  26. [37]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 b . http://arxiv.org/abs/2303.16634 G-eval: Nlg evaluation using gpt-4 with better human alignment

    2023 arXiv

  27. [38]

    Sri Raghava Muddu, Rupasai Rangaraju, Tejpalsingh Siledar, Swaroop Nath, Pushpak Bhattacharyya, Swaprava Nath, Suman Banerjee, Amey Patil, Muthusamy Chelliah, Sudhanshu Shekhar Singh, and Nikesh Garera. 2024. http://arxiv.org/abs/2406.10886 Distilling opinions at scale: Increm...

    2024 arXiv

  28. [39]

    OpenAI . 2023. ChatGPT (August 3 Version) . https://chat.openai.com

    2023

  29. [40]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . ArXiv, abs/2303.08774

    2023 arXiv

  30. [41]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  31. [42]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. http://arxiv.org/abs/1910.10683 Exploring the limits of transfer learning with a unified text-to-text transformer

    2023 arXiv

  32. [43]

    Yuchen Shen and Xiaojun Wan. 2023. http://arxiv.org/abs/2310.18122 Opinsummeval: Revisiting automated evaluation for opinion summarization

    2023 arXiv

  33. [44]

    Tejpalsingh Siledar, Swaroop Nath, Sankara Sri Raghava Ravindra Muddu, Rupasai Rangaraju, Swaprava Nath, Pushpak Bhattacharyya, Suman Banerjee, Amey Patil, Sudhanshu Shekhar Singh, Muthusamy Chelliah, and Nikesh Garera. 2024 a . http://arxiv.org/abs/2402.11683 One prompt to ru...

    2024 arXiv

  34. [45]

    Tejpalsingh Siledar, Rupasai Rangaraju, Sankara Sri Raghava Ravindra Muddu, Suman Banerjee, Amey Patil, Sudhanshu Shekhar Singh, Muthusamy Chelliah, Nikesh Garera, Swaprava Nath, and Pushpak Bhattacharyya. 2024 b . http://arxiv.org/abs/2404.05243 Product description and qa ass...

    2024 arXiv

  35. [46]

    Yoshihiko Suhara, Xiaolan Wang, Stefanos Angelidis, and Wang-Chiew Tan. 2020. https://doi.org/10.18653/v1/2020.acl-main.513 O pinion D igest: A simple framework for opinion summarization . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisti...

    2020 doi

  36. [47]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex...

    2024 arXiv

  37. [48]

    Rush, and Thomas Wolf

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023. http://arxiv.org/abs/2310.16944 Zephy...

    2023 arXiv

  38. [49]

    Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/7a98af17e63a0ac09ce2e96d03992fbc-Paper.pdf Neural discrete representation learning . In Advances in Neural Information Processing Systems, volume 30. Curr...

    2017

  39. [50]

    Xingchen Wan, Ruoxi Sun, Hanjun Dai, Sercan Arik, and Tomas Pfister. 2023. https://doi.org/10.18653/v1/2023.findings-acl.216 Better zero-shot reasoning with self-adaptive prompting . In Findings of the Association for Computational Linguistics: ACL 2023, pages 3493--3514, Toro...

    2023 doi

  40. [51]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. https://doi.org/10.18653/v1/2023.newsum-1.1 Is C hat GPT a good NLG evaluator? a preliminary study . In Proceedings of the 4th New Frontiers in Summarizatio...

    2023 doi

  41. [52]

    Lu Wang and Wang Ling. 2016. Neural network-based abstract generation for opinions and arguments. arXiv preprint arXiv:1606.02785

    2016 arXiv

  42. [53]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. http://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models

    2023 arXiv

  43. [54]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  44. [55]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  45. [56]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. http://arxiv.org/abs/2106.11520 Bartscore: Evaluating generated text as text generation

    2021 arXiv

  46. [57]

    Haoyu Zhang, Jingjing Cai, Jianjun Xu, and Ji Wang. 2019. https://doi.org/10.18653/v1/K19-1074 Pretraining-based natural language generation for text summarization . In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 789--797, Hong ...

    2019 doi

  47. [58]

    Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020 a . http://arxiv.org/abs/1912.08777 Pegasus: Pre-training with extracted gap-sentences for abstractive summarization

    2020 arXiv

  48. [59]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020 b . http://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert

    2020 arXiv

  49. [60]

    Chao Zhao and Snigdha Chaturvedi. 2020. https://doi.org/10.1609/aaai.v34i05.6512 Weakly-supervised opinion summarization by leveraging external information . Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9644--9651

    2020 doi

  50. [61]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  51. [62]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.