{"id":"007ce1aa-e001-4171-92fb-3262bf3d5a40","arxiv_id":"2607.10825","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Multidimensional classification plus Knapsack, Knapsack-KL, and KDE stratified sampling yields token-efficient LLM opinion summaries that preserve topic coverage and semantic fidelity better than random subsets.","lead":"A framework classifies opinions by sentiment, topic, and emotion, then uses stratified sampling to pick a small balanced subset before LLM summarization. It cuts input tokens by roughly 97% while matching full-corpus summary meaning better than random selection on product, hotel, and political posts.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Full-context LLM summaries are an unreliable gold standard for the cosine-similarity claim, especially given the paper's own citation that large contexts degrade quality.","rationale":"The reader correctly isolates the evaluation design as the softest load-bearing point. The engineering contribution (facet-aware Knapsack/KL/KDE selection under token budgets, public code, multi-domain results) is real and the algorithms are clearly stated. However, the headline numbers that justify “semantic-preserving” and “outperforming \to standard LLM summarization” are measured against an LLM full-context gold that the paper’s own citation [29] and token counts render suspect. Strengthening with human references or non-LLM baselines would convert the claim from circular to externally validated; without that, CONDITIONAL remains the right verdict. No stronger internal inconsistency was found.","tokens_in":14212,"tokens_out":518,"duration_ms":7866,"concrete_test":"On a fixed subset of 20–30 Amazon/Tripadvisor instances, obtain independent human reference summaries (or at least human-ranked topic lists) of the full opinion sets. Recompute ModernBERT cosine similarity of stratified vs. random sample summaries against the human references (instead of GPT-5 full-context). If the stratified advantage over random shrinks below ~0.05 or absolute scores fall below 0.75, the central semantic-preservation claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim rests on summary-level cosine similarity (ModernBERT embeddings) between sample-based summaries and full-corpus summaries (Section 4.2.2, Figures 4(a)–4(c)). The paper treats the full-context GPT-5 summary as the semantic reference that stratified samples of N≈20 approach (0.83–0.88). Yet the same paper cites [29] that very large contexts can degrade summary quality, and full sets are described as exceeding 28k–100k tokens. If the full-context summary is itself biased, incomplete, or lower-quality, high cosine similarity only shows that the sample reproduces that flawed reference, not that it preserves the true diversity of the corpus. Topic-coverage F1 is likewise measured against ChatGPT-Thinking free-form topics rather than human labels (Section 4.2.1). Thus the quantitative backbone of “semantic preservation while beating random” is circular: both the gold and the metric are LLM-derived under conditions the authors themselves flag as problematic.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes a corpus-level framework for LLM-based opinion summarization that first annotates opinions with multidimensional probabilistic facets (sentiment, emotion, BERTopic topics, and optional domain facets) and then selects a compact subset under a fixed size N via three stratified samplers—relevance-constrained Knapsack, KL-regularized Knapsack-KL (Eqs. 1–2, Algorithms 1–2), and KDE—before facet-aware prompting. The central claim is that these distribution-aligned subsets (N≈20) preserve topical structure and yield summaries whose ModernBERT cosine similarity to full-corpus GPT-5 summaries substantially exceeds random sampling, while cutting input tokens by roughly 97% on Amazon product reviews, Tripadvisor hotel reviews, and X/Twitter political posts. An ablation on semantic dimensions and a public implementation are provided.","tokens_in":14516,"tokens_out":1189,"duration_ms":21290,"significance":"If the semantic-preservation claim holds under stronger evaluation, the work is practically significant: it offers a clear, implementable alternative to brute-force long-context ingestion or query-specific RAG for large opinion corpora, with explicit multi-facet balance and documented token savings. Strengths include formalized sampling algorithms with pseudocode, three heterogeneous domains, a dimension ablation (Table 2), efficiency trade-off discussion, and a public GitHub release. These elements make the contribution reproducible and useful for applied opinion mining even if some evaluation choices need tightening. The advance over the authors’ prior conference version (broader samplers, multi-domain protocol, token-focused metrics) is real but incremental.","major_comments":[{"comment":"Section 4.2.2 and Figures 4(a)–4(c): the primary semantic-preservation metric is cosine similarity of sample-based summaries to full-corpus GPT-5 summaries. The paper itself cites evidence that very large contexts can degrade summary quality [29] and reports full sets of 28k–100k+ tokens. Treating the full-context output as the gold standard therefore risks measuring fidelity to a potentially biased or incomplete reference rather than to the true opinion distribution. This is load-bearing for the abstract’s “semantic preservation” claim and should be addressed by (i) human preference or aspect-coverage judgments, (ii) an independent reference (e.g., extractive multi-document baselines or human-written summaries on a subset), or (iii) explicit analysis showing that full-context quality does not degrade on these corpora.","section":null},{"comment":"Abstract and Section 4: the abstract states that the method “consistently outperform[s] traditional AI-based and standard LLM summarization baselines,” yet the reported experiments compare mainly against random sampling and full-context LLM input (plus internal sampler variants). No classical multi-document summarizers (e.g., LexRank, MMR, clustering-based selection), no sentiment-only or single-facet filters, and no standard long-context or map-reduce LLM pipelines appear as quantitative baselines. The claim as written is not supported by the experimental section and should either be narrowed or backed by those comparisons.","section":null},{"comment":"Section 4.2.1: ground-truth topics for coverage F1 are free-form themes extracted by ChatGPT-Thinking and matched via ModernBERT embeddings. Without human topic labels or inter-annotator agreement on a sample, both the absolute F1 numbers and the ranking of samplers remain LLM-dependent. A modest human-validated topic set (or at least a sensitivity check against an alternative topic model) is needed to underwrite the “topic coverage” half of the main claim.","section":null}],"minor_comments":[{"comment":"Section 3.2: the relevance score (top-k BERTopic term counts, normalized) is simple; a short sensitivity analysis on k, or comparison to embedding-based relevance, would strengthen the design justification.","section":null},{"comment":"Algorithm 2 / Eq. (1): the linear schedule for α(i) is stated but not ablated; reporting results for fixed α or alternative schedules would clarify robustness of Knapsack-KL and KDE.","section":null},{"comment":"Figure 2 and the hotel walkthrough are helpful; adding analogous distribution plots for Amazon and X/Twitter (even in appendix) would make multi-domain balance more transparent.","section":null},{"comment":"Table 1 averages across domains; per-domain rows (or appendix tables) would show whether KDE’s edge is consistent or driven by one corpus.","section":null},{"comment":"Related work: coverage of recent long-context and opinion-summarization LLM papers is thin; a few additional 2024–2025 citations would better situate the contribution.","section":null},{"comment":"Minor typos/notation: “KullLei” in figure legends vs “Knapsack-KL” in text; “T opic” spacing artifacts in subsection titles; ensure consistent N vs sample size wording.","section":null}],"recommendation":"major_revision","confidential_remarks":"The evaluation circularity flagged by the reader is real and load-bearing; I would not accept without either human evaluation or a non-LLM reference for the cosine claim. Novelty relative to the authors’ own ECML PKDD 2025 conference paper is incremental (mainly broader samplers and multi-domain token-focused metrics); the journal version is still a reasonable extension if the evaluation gaps are closed. Scope fit for a solid applied NLP / opinion-mining venue is good."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a clean engineering extension of the authors’ own ECML PKDD 2025 paper. What is new is the formal family of three distribution-aware samplers—Knapsack, Knapsack-KL with the α-scheduled score in (1)–(2), and KDE—plus multi-domain runs (Amazon, Tripadvisor, X/Twitter) and an ablation on sentiment/topic/emotion facets. They ship algorithms with pseudocode, public code, and a clear token-budget story: at N≈20 you get topic-coverage F1 and ModernBERT cosine similarity to full-context summaries that beat random while using ~1–2% of the tokens.\n\nThat core claim holds up as an applied result. Stratified selection under multi-facet constraints is not brand-new in multi-document summarization, but they make it concrete for opinion corpora, show KDE edges the others on fidelity while Knapsack is the practical speed/quality trade-off, and the ablation confirms that adding topic and emotion helps. The hotel walk-through and figures make the distribution-matching idea easy to see.\n\nSoft spots are real but proportionate. The abstract’s “traditional AI-based and standard LLM baselines” language oversells what is actually reported—mostly random sampling and full-context GPT-5. Topic ground truth comes from ChatGPT-Thinking and the cosine target is the full-context summary itself; the paper even cites [29] that large contexts can degrade quality, so the gold is imperfect. That weakens the “semantic preservation” framing more than the practical “beats random under a token budget” claim. No uncertainty estimates, free parameters (N, α schedule, top-k, histogram settings) are not deeply swept. None of this breaks the engineering result.\n\nWho it is for: people building review/social analytics pipelines who need cheaper LLM summarization without pure majority bias. A serious referee should see it; it is reproducible and useful once baselines and non-LLM evaluation are tightened. I would engage, cite the samplers if I am doing token-constrained opinion work, and send it to review rather than desk-reject.","headline":"Solid journal extension of their ECML work: three formalized stratified samplers cut tokens ~97% and beat random on multi-domain opinion corpora, with public code; evaluation gold standards are LLM-derived and baselines thinner than the abstract claims.","tokens_in":15124,"tokens_out":549,"would_cite":true,"duration_ms":7321,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Stratified sampling of opinion facets lets LLMs summarize huge review corpora with ~97% fewer tokens while keeping viewpoint diversity.","keywords":["Large Language Models","Opinion Summarization","Stratified Sampling","Token Efficiency","Semantic Preservation","Review Mining","Multidimensional Classification"],"falsifier":"If, on the same three datasets, random samples of equal size matched or beat the stratified methods on both topic-coverage F1 and ModernBERT cosine similarity to a carefully constructed human or multi-model reference summary, the claimed advantage of distribution-aware selection would collapse.","tokens_in":15096,"feed_emoji":"📉","tokens_out":551,"duration_ms":7396,"temperature":0.7,"pith_summary":"Huge collections of product reviews, hotel feedback, and social posts are redundant, imbalanced, and expensive to feed whole into a large language model. This paper argues that the right fix is not a bigger context window or retrieval, but a corpus-level selection step: first label every opinion on multiple semantic facets (sentiment, topics, emotion, and domain-specific labels), then use stratified samplers to pick a small, distribution-matched subset before the LLM ever sees the text. On Amazon electronics reviews, Tripadvisor New York hotels, and 2024 election posts from swing-state Twitter users, samples of roughly twenty opinions produce summaries whose topic coverage and embedding similarity to full-corpus summaries far exceed random selection, while cutting input tokens by about 97 percent. The practical payoff is scalable, viewpoint-balanced opinion digests that stay faithful to minority as well as majority voices without paying full-corpus token cost.","feed_headline":"20 reviews replace 50k tokens and still keep viewpoint balance","feed_subtitle":"Facet-aware sampling cuts LLM input cost by ~97% on Amazon, hotels, and Twitter while matching full-corpus summaries","key_machinery":"Multidimensional stratified sampling: each opinion is first given probabilistic labels on sentiment, topics, emotion (and optional domain facets); three strategies (relevance-constrained Knapsack, KL-regularized Knapsack-KL, and density-based KDE) then select a fixed-size subset that preserves those class distributions under a hard token budget.","core_discovery":"A compact subset of opinions selected by multidimensional stratified sampling (Knapsack, Knapsack-KL, or KDE) can be summarized by an LLM so that topic coverage and summary-level cosine similarity stay close to the full-corpus summary, while token usage falls by roughly two orders of magnitude.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Stratified sampling: 20 reviews match full-corpus balance at 100x fewer tokens","Facet-aware subsets cut LLM input ~97% yet keep topic coverage and cosine sim","Knapsack-KL picks compact opinions that preserve semantic summary fidelity","Multidim sampling lets LLM summaries retain viewpoints with two-order token drop","Select by sentiment and topics: balanced summaries from tiny opinion subsets"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The evaluation treats the LLM summary of the entire unfiltered corpus as a reliable semantic gold standard, even though very large contexts can themselves degrade quality.","fun_headline_variants_meta":{"raw":{"variants":["Stratified sampling: 20 reviews match full-corpus balance at 100x fewer tokens","Facet-aware subsets cut LLM input ~97% yet keep topic coverage and cosine sim","Knapsack-KL picks compact opinions that preserve semantic summary fidelity","Multidim sampling lets LLM summaries retain viewpoints with two-order token drop","Select by sentiment and topics: balanced summaries from tiny opinion subsets"]},"model":"grok-4.5","effort":"low","cost_usd":0.005482,"raw_usage":{"total_tokens":1455,"prompt_tokens":716,"num_sources_used":0,"completion_tokens":83,"cost_in_usd_ticks":54820000,"prompt_tokens_details":{"text_tokens":716,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":656,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":716,"tokens_out":83,"duration_ms":9387,"temperature":1.0,"reasoning_tokens":656,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T08:57:10.901910+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"If, on the same three datasets, random samples of equal size matched or beat the stratified methods on both topic-coverage F1 and ModernBERT cosine similarity to a carefully constructed human or multi-model reference summary, the claimed advantage of distribution-aware selection would collapse.","supporting_citations":[],"review_version":1}