QQSUM: A Novel Task and Model of Quantitative Query-Focused Summarization for Review-based Product Question Answering

An Quang Tang; Minh Ngoc Dinh; Xiuzhen Zhang; Zhuang Li

REVIEW 4 major objections 5 minor 12 references

QQSUM: A Novel Task and Model of Quantitative Query-Focused Summarization for Review-based Product Question Answering

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces Quantitative Query-Focused Summarization (QQSUM), in which product-question answers are bullet-point key opinions each paired with how many review comments support it, and proposes QQSUM-RAG, a retrieval-augmented…

desk verdict A genuinely new task and usable dataset, but the quantification results rest on a shaky LLM-as-judge loop and need human-grounded test labels before the strong claims can be trusted. read the letter →

arxiv 2506.04020 v1 pith:BGTCNUWF submitted 2025-06-04 cs.CL

An Quang Tang , Xiuzhen Zhang , Minh Ngoc Dinh , Zhuang Li This is my paper

classification cs.CL

keywords QuantitativeQuery-FocusedSummarizationKeyPointAnalysisRetrieval-AugmentedGenerationProductQuestionAnsweringOpinionPrevalenceQuantificationFew-shotLearningE-commercereviews

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a new task, Quantitative Query-Focused Summarization (QQSUM), in which an answer to a product question is a bullet-point list of key opinion points, each paired with the number of review comments that support it. The authors argue that standard review-based product question answering produces a single dominant opinion and misses the diversity of customer viewpoints, and that existing Key Point Analysis methods summarize opinions without tying them to a specific query. To solve this, they propose QQSUM-RAG, a retrieval-augmented generation model that clusters query-relevant comments by opinion before generating key points, and trains the retriever and generator jointly from a small set of human-LLM annotations. They report that QQSUM-RAG substantially outperforms RAG and Key Point Analysis baselines on both textual quality and prevalence quantification. If correct, this gives e-commerce platforms a way to answer subjective questions with summaries that reflect the distribution of customer opinions rather than a single majority view.

What carries the argument

The load-bearing mechanism is the KP-oriented retrieval stage, which turns the retriever into an opinion clusterer: comments are embedded with Contriever, ranked by dot-product similarity to the query, and greedily grouped into clusters whose average cosine similarity exceeds a threshold; each cluster is meant to correspond to one key point. The generation stage then uses a 'next-KP' objective where the LLM produces each bullet point conditioned on the previous ones, so later key points are forced to be non-overlapping. A co-training loss ties the two stages: the retriever minimizes squared error against the centroid of the annotated comment-cluster for each comment, while the LLM's generation loss is back-propagated into the retriever through a perplexity-distillation term, aligning retrieved clusters with the key points the generator actually produces.

What would settle it

Replace GPT-4-o-mini's labels on the full test set with human annotations from the same MTurk protocol, or compare QQSUM-RAG's outputs with those of PAKPA using a blind human pairwise evaluation on all 136 questions; if the quantified advantage disappears or reverses, the reported superiority is an evaluation artifact rather than a genuine difference in summary quality.

Watch

Extended reading notes

Core claim

QQSUM-RAG's central claim is that the diversity and quantitative accuracy of query-focused opinion summaries depend on making retrieval opinion-aware: instead of feeding a large language model a flat list of retrieved reviews, the model first groups comments into clusters that each correspond to a distinct opinion relevant to the query, then generates one key point per cluster, using already generated key points as context to avoid redundancy. The framework co-trains a Contriever retriever and a small open-source LLM (Mistral-7B or Vicuna-7B) with a loss that combines a cluster-alignment retrieval loss, a perplexity-distillation signal from the LLM to the retriever, and a next-key-point generation loss. The authors claim this yields up to 2.11 times higher textual similarity with reference key points (ROUGE-1 0.256 vs 0.121) and up to 67.12% better quantification performance (F1 0.792 vs 0.619, QuantErr 4.24 vs 6.68) over the strongest KPA baseline, PAKPA.

Load-bearing premise

The automatic evaluation treats GPT-4-o-mini's extracted key points and comment-matching labels as ground truth for the entire 136-question test set, even though the annotator was validated against human judgments on only five queries.

Editorial extensions

If this is right

QQSUM establishes a new evaluation paradigm for product question answering, where answers are judged not only on fluency but on whether they enumerate distinct opinions and count how many comments support each one.
A small set of curated annotations (34 instances) suffices to train a retrieval-augmented opinion summarizer that outperforms strong zero- and few-shot RAG baselines, suggesting the co-training objective transfers well.
Clustering before generation is what enables diversity: ablations show that removing the next-KP context drops quality below RAG baselines, while KP-oriented retrieval alone still preserves quantification accuracy.
The framework generalizes across 17 Amazon product categories and works with different open-source LLMs, with Mistral-7B generally outperforming Vicuna-7B.
Quantifying opinion prevalence can reduce hallucination: QQSUM-RAG matches or exceeds PAKPA on factual alignment (AlignScore 0.749) while covering far more comments (recall 0.869 vs 0.520).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension would be to apply the same co-trained clustering-plus-generation recipe to other multi-perspective summarization tasks, such as controversial political questions or medical treatment decisions, where prevalence counts could replace review counts.
The gold-score perplexity distillation implicitly assumes that lower LLM perplexity on a comment given a cluster indicates better alignment with the key point; one could test this directly by correlating perplexity drops with human match judgments, a check the paper does not perform.
If GPT-4-o-mini's annotation bias is real, the reported margins might be an upper bound; a practical safeguard for future work is to validate LLM annotators on a larger stratified sample, especially for categories like 'Controversial' questions with lower prevalence.
The clustering threshold (cosine similarity 1.2) is set empirically; the paper does not report sensitivity to this value, so a robustness sweep would clarify whether the method's advantage is robust or tuned to the evaluation set.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing to know: this paper introduces a genuinely new task -- quantitative query-focused summarization for PQA -- and builds a usable dataset (AMAZONKP) from AmazonQ&A. The model is a reasonable integration of existing ideas (clustering + RAG + next-KP generation), not a radical new architecture. The human Bradley-Terry evaluation on 5 queries suggests the generated summaries are better on coverage, validity, and sentiment than the baselines. That part I believe.

The soft spot is exactly where the stress-test points: the quantification results in Table 5 are computed entirely against GPT-4-o-mini labels. The same model family produced the reference KPs (Stage 1), the comment-KP match annotations used as gold for training and evaluation, and the evaluator that scores predicted matches. Human validation is tiny: 5 queries for KP extraction, 5 queries for comment-KP matching (Pearson r=0.647). That correlation is moderate, not strong, and it doesn't cover the full test set. So the headline numbers -- F1 0.792 vs 0.619, QuantErr 4.24 vs 6.68 -- are plausible but not robust. The gap could be real, or it could be inflated by shared LLM perspective, since QQSUM-RAG's clustering produces KPs that are structurally closer to GPT-4-o-mini's notion of a match.

There are other weaknesses. The test set is a sample of 136 of 2516 questions with no error bars or significance tests, so "significantly outperforms" is doing too much work. The textual reference KPs are extracted from community answers, not from the reviews, so ROUGE/BERTScore comparisons measure fidelity to a proxy gold, as the limitations section acknowledges. None of this kills the central contribution, but it does mean the paper's strong quantitative claims should be read as conditional.

What's solid: the task definition, the dataset curation pipeline with human-LLM collaboration, the ablation showing next-KP generation helps, and the honest limitations paragraph. The paper ships code. The citation pattern looks fine; PAKPA is the natural prior baseline from the same group.

This is a paper for people working on opinion summarization, PQA, and RAG evaluation. It deserves a real referee, but the referee should push for human-annotated gold labels for at least a subset of the test set, error bars or significance tests, and a comparison that controls for the shared-LLM evaluation bias. With those additions the central claim could be accepted. My recommendation: send it to peer review, not desk reject, but expect major revision.

Referee Report

4 major / 5 minor

Summary. This paper introduces Quantitative Query-Focused Summarization (QQSUM), a task in which a system must produce a bullet-point summary of key points (KPs) from product reviews and quantify each KP's prevalence relative to a user query. The authors construct AMAZONKP by curating subjective questions from AmazonQ&A, using GPT-4-o-mini with MTurk validation to extract KPs from community answers and to match comments to KPs for a 34-query training set. Their model, QQSUM-RAG, extends Atlas with a KP-oriented retriever that clusters comments, a jointly trained LLM generator with a next-KP generation objective, and few-shot co-training. Experiments compare QQSUM-RAG with RAG, prompt-LLM, and KPA baselines on retrieval, textual quality, and quantification, reporting substantial gains (e.g., 0.792 vs. 0.619 F1 and QuantErr 4.24 vs. 6.68 over PAKPA in Table 5).

Significance. If the empirical claims held, QQSUM would be a useful task formulation for review-based PQA, and the modeling idea of joint KP-oriented retrieval and iterative KP generation is a plausible contribution. Strengths include the release of code, a carefully described human-LLM annotation pipeline, and ablation experiments isolating the clustering and next-KP generation components. The main limitation is that the headline quantification results rest on an LLM judge that is also used to create the reference annotations and evaluate LLM-generated outputs; only a 5-query human validation with moderate correlation (r=0.647) supports that judge. The paper's Limitations section also concedes that the reference KPs are extracted from community answers rather than reviews, so the automatic textual evaluation measures fidelity to a proxy. These concerns do not invalidate the task or method, but they make the 'significantly outperforms' claim not yet robustly supported.

major comments (4)

[4.2.2, Table 5, Appendix J] The KP-comment matching and QuantErr results in Table 5 are computed from gpt-4-o-mini match labels, and this same model is also used to extract reference KPs and to post-process system outputs. The only human validation of the judge (Appendix J) covers 5 queries, with Pearson r=0.647 and accuracy 0.807. Because all evaluated systems are LLM-based and QQSUM-RAG's clusters are built from embeddings that may align with the judge's notion of similarity, the reported F1 and QuantErr advantages over PAKPA (0.792 vs. 0.619; 4.24 vs. 6.68) could be an artifact of judge bias rather than genuine quantification accuracy. Please report human-annotated comment-KP matches on a substantially larger test subset, for all systems, and compute prevalence errors against those human labels; also report agreement statistics and per-query variance.
[3.3, 4.2.1, Limitations] The reference KPs used in the automatic textual evaluation are extracted from AmazonQ&A community answers, not from the product reviews that the systems summarize. The Limitations section itself acknowledges that KPs 'might not fully in line with viewpoints in reviews' and that review opinions may not cover community-answer opinions. Therefore Table 3's ROUGE/BERTScore/BLEURT/G-Eval comparisons measure fidelity to a proxy gold that is not the system's input domain. The human Bradley-Terry evaluation in Table 4 is on only 5 queries and covers only textual dimensions, so it cannot substitute for a human-grounded quantification evaluation. Please either add human evaluation of KP textual quality on a larger sample or explicitly re-frame the task as summarizing community answers rather than reviews.
[4.2, Tables 3 and 5] The automatic evaluation uses only 136 of 2,516 test questions (8 per category), with no confidence intervals, bootstrap estimates, or significance tests. The text repeatedly uses 'significantly outperforms' (e.g., Section 1, Section 4.3.3), but the reported improvements are point estimates on a small sample. Please provide paired significance tests (or effect sizes with CIs) across queries and clarify how the 8 questions per category were sampled.
[3.2.1, Eq. (1)-(3)] The joint training objective is underspecified: Eq. (1) includes gold_score as a 'Perplexity Distillation loss' with no explicit formula, and the values/selection procedure for the damping factor d, the clustering threshold Lambda, the comment selection threshold, the LoRA rank, and the number of training steps are not reported. Since the method's few-shot behavior may be sensitive to these hyperparameters, the paper should report exact values and a sensitivity analysis for the main results.

minor comments (5)

[Throughout] There are several typos, including 'Apppendix E' (Section 3.3), 'factal' (Appendix A), 'valiation' (Table 8 caption), 'gp4-o-mini' (Appendix B), 'Retreiver' (Section 4.3.2), 'intepretation' (Appendix B), and 'aggegrated' (Section 4.1).
[4.2.1] G-EVAL scores are said to be scaled from 1-5 to 0-1, but the scaling formula is not given; please specify whether it is (score-1)/4 or another mapping.
[Table 4] The Bradley-Terry scores are not defined; please state whether they are expected win probabilities or ranking coefficients, and how ties/no-preference responses were handled.
[4.2.1, Eq. (6)] RD is undefined when a generated summary has fewer than two KPs; please specify the convention for degenerate cases.
[3.3, Appendix E] Stage 2 says LLM-matched pairs are validated by three MTurk workers, but Appendix E says 'at least 60% of the annotators had to agree'; please reconcile these thresholds and state the final aggregation rule.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: held-out test set and human-validated LLM judge keep the central comparison independent.

full rationale

The claimed chain—KP-oriented retrieval, cluster-based KP generation, and quantification—is not circular by the paper's own equations. Training uses 34 annotated queries, while evaluation is on a held-out test set of 136 queries whose Stage-1 KPs come from gold community answers. The retriever loss (Eq. 2) and generator loss (Eq. 3) are optimized against human-validated comment–KP annotations, not against the test-time evaluator. The quantification evaluation (Table 5) is computed by prompting gpt-4-o-mini, but Appendix J validates that judge against MTurk on 5 test queries (Pearson r=0.647, accuracy 0.807), and the paper explicitly acknowledges in Limitations that community-answer KPs may not fully align with review opinions. Self-citations to Tang et al. (2024b) establish the PAKPA baseline, which is empirically compared rather than invoked to force the result. No fitted hyperparameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work. The closest concern—shared LLM annotation and evaluation—is a measurement-validity limitation, not a definitional reduction, because the model outputs are not the same tokens as the reference labels and the test queries are disjoint from the training queries.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the validity of LLM-generated labels and hand-set thresholds. The model itself is a standard RAG pipeline with clustering; no new theoretical entities are introduced.

free parameters (4)

comment selection threshold = 1
s(x,r_j) >= 1 is used to select relevant comments for retrieval, noted as set empirically in Section 3.2.1.
clustering threshold lambda = 1.2
Average cosine similarity threshold for assigning a comment to an existing cluster, set empirically based on cluster quality in Section 3.2.1.
damping factor d = not specified
Weights the retrieval loss against the generation loss in Eq. (1); the value is not reported, affecting the balance between retriever and generator updates.
LoRA rank = not specified
Low-Rank Adaptation rank for parameter-efficient fine-tuning, mentioned in Section 4 but not specified.

assumptions (5)

domain assumption AmazonQ&A community answers serve as an adequate gold source of diverse opinions for the QQSUM task.
The dataset is built on the premise that user-provided answers to product questions represent the diversity of opinions to be summarized, as stated in Section 3.3.
domain assumption GPT-4-o-mini's key point extraction and comment-KP matching annotations are reliable enough to serve as ground truth.
The entire automatic evaluation and part of the training supervision depend on LLM-generated labels, with validation only on small samples (Appendices C and J).
domain assumption The dot product of Contriever embeddings is a valid measure of semantic similarity for relevance and clustering.
Used throughout the retriever and clustering algorithm, Section 3.2.1.
domain assumption A training set of 34 instances is sufficient to learn the retriever and generator for a few-shot setting.
The paper relies on few-shot learning with only 2 queries per category for supervision, Section 3.3.
domain assumption MTurk annotator judgments, with quality filters, are a valid gold standard for validating LLM annotations.
Used to validate KP extraction and comment-KP matching, Sections 3.3 and Appendix J.

how reviews work

0 comments

Cite this review

Pith. "Pith review of QQSUM: A Novel Task and Model of Quantitative Query-Focused Summarization for Review-based Product Question Answering." pith.science (2026). https://pith.science/paper/BGTCNUWF

@misc{pith2026250604020,
  author       = {Pith},
  title        = {Pith review of: QQSUM: A Novel Task and Model of Quantitative Query-Focused Summarization for Review-based Product Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BGTCNUWF}},
  note         = {Machine review of arXiv:2506.04020}
}

read the original abstract

Review-based Product Question Answering (PQA) allows e-commerce platforms to automatically address customer queries by leveraging insights from user reviews. However, existing PQA systems generate answers with only a single perspective, failing to capture the diversity of customer opinions. In this paper we introduce a novel task Quantitative Query-Focused Summarization (QQSUM), which aims to summarize diverse customer opinions into representative Key Points (KPs) and quantify their prevalence to effectively answer user queries. While Retrieval-Augmented Generation (RAG) shows promise for PQA, its generated answers still fall short of capturing the full diversity of viewpoints. To tackle this challenge, our model QQSUM-RAG, which extends RAG, employs few-shot learning to jointly train a KP-oriented retriever and a KP summary generator, enabling KP-based summaries that capture diverse and representative opinions. Experimental results demonstrate that QQSUM-RAG achieves superior performance compared to state-of-the-art RAG baselines in both textual quality and quantification accuracy of opinions. Our source code is available at: https://github.com/antangrocket1312/QQSUMM

Figures

Figures reproduced from arXiv: 2506.04020 by the authors.

**Figure 2.** The training architecture of the QQSUM-RAG framework. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Illustration of the human-LLM collaborative [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 11 canonical work pages

[1]

Identify unique opinions from the answers of the given question

work page
[2]

good" or “bad

AlignScore: Evaluating factual consistency with a unified alignment function. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328–11348, Toronto, Canada. Association for Computational Linguistics. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020a. Pegasus: Pre-traini...

work page 2020
[3]

Assess how well the key point covers the main points of the source key point, and how much irrelevant or redundant information it contains

work page
[4]

Scenario-based

Based on the question content and the amount of opinions in the question's answer, identify the question's type. Note that you must briefly explain why the question is opinionated or factual before giving the final decision. Below are some few−shot examples: Questions: How well does it work with wireless charging Answers: ['Unfortunately with this case in...

work page 2021
[5]

In every answer from the list, extract all possible key point candidates

work page
[6]

Below are some few−shot examples: Questions: Can I use these for running/working out? Do they handle sweat? Answers: ['I have seen other people using these for running/working out

From the extracted list of key point candidates, generate a list of only general and non−overlapping key points that are relevant and can answer the input opinionated question. Below are some few−shot examples: Questions: Can I use these for running/working out? Do they handle sweat? Answers: ['I have seen other people using these for running/working out....

work page
[10]

Listing 5: Zero-shot prompt for G-EVALredundancy evaluation of generated KPs, supportingRDcalculation

Assign a relevance score from 1 to 5. Listing 5: Zero-shot prompt for G-EVALredundancy evaluation of generated KPs, supportingRDcalculation. You will be given one key point, short salient sentence, written to describe user opinion on a product. Your task is to rate the summary on one metric. Please make sure you read and understand these instructions care...

work page
[11]

Read the key point and the source key point carefully

work page

Show all 12 references

[12]

Compare the key point to the source key point and identify the main points
[13]

Assess how much redundant opinion and information the key point covers that overlap with the source key point
[14]

I came for a company event

Assign a redundancy score from 1 to 5. negative). This would exclude sentences like “I came for a company event”. • INFORMATIVENESS: The key point in the summary should discuss should discuss some aspects of the reviewed product and contain useful information. Any key point th...

2021
[2023]

InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 11951–11964, Toronto, Canada

Product question answering in E-commerce: A survey. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 11951–11964, Toronto, Canada. Association for Computational Linguistics. Roni Friedman, Lena Dankin, Y...

2021 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Identify unique opinions from the answers of the given question

work page

[2] [2]

good" or “bad

AlignScore: Evaluating factual consistency with a unified alignment function. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328–11348, Toronto, Canada. Association for Computational Linguistics. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020a. Pegasus: Pre-traini...

work page 2020

[3] [3]

Assess how well the key point covers the main points of the source key point, and how much irrelevant or redundant information it contains

work page

[4] [4]

Scenario-based

Based on the question content and the amount of opinions in the question's answer, identify the question's type. Note that you must briefly explain why the question is opinionated or factual before giving the final decision. Below are some few−shot examples: Questions: How well does it work with wireless charging Answers: ['Unfortunately with this case in...

work page 2021

[5] [5]

In every answer from the list, extract all possible key point candidates

work page

[6] [6]

Below are some few−shot examples: Questions: Can I use these for running/working out? Do they handle sweat? Answers: ['I have seen other people using these for running/working out

From the extracted list of key point candidates, generate a list of only general and non−overlapping key points that are relevant and can answer the input opinionated question. Below are some few−shot examples: Questions: Can I use these for running/working out? Do they handle sweat? Answers: ['I have seen other people using these for running/working out....

work page

[7] [10]

Listing 5: Zero-shot prompt for G-EVALredundancy evaluation of generated KPs, supportingRDcalculation

Assign a relevance score from 1 to 5. Listing 5: Zero-shot prompt for G-EVALredundancy evaluation of generated KPs, supportingRDcalculation. You will be given one key point, short salient sentence, written to describe user opinion on a product. Your task is to rate the summary on one metric. Please make sure you read and understand these instructions care...

work page

[8] [11]

Read the key point and the source key point carefully

work page