{"id":"365a2ceb-5970-48bf-9aad-6cba219527fa","arxiv_id":"1908.00648","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"An unsupervised pipeline uses phrase mining and a topic-viewpoint model with author interactions to extract a pro/con table of sentential reasons from online debates.","lead":"The paper builds an unsupervised pipeline that finds and clusters the main reasons people give for and against a controversial issue in online debates. It tests the pipeline on abortion, gay rights, and other debates, and reports improvements over existing summarization methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim of 'significant improvement' is unsupported by any significance test: Tables 3 and 4 report only averages and medians over 5–10 runs, with no variance or confidence intervals.","rationale":"The reader's verdict is CONDITIONAL, and my read does not move that verdict: the paper is plausible but the evaluation is not yet robust enough to fully support the headline. However, I identify a different load-bearing concern than the reader's weakest_assumption. The reader highlights the Rb=1 assumption as the weakest point. That is a real modeling limitation, but it is partially mitigated by the dataset statistics reported in Table 2 (67–78% rebuttals) and by the paper's own concluding caveat that other platforms may differ. The Rb assumption threatens generalization to new platforms, not necessarily the validity of the results on the evaluated debate datasets. In contrast, the phrase 'significant improvement' is a direct statistical claim about the evaluated results, and the paper provides no significance tests, error bars, or variance information anywhere. The difference between PhAITV and TAM or LAM_LEX on some metrics is large, but without knowing the run-to-run variance, 'significant' is not established. A paired test on the existing runs, or a rerun with more seeds, would settle this directly. If the differences survive a significance test, the central claim is supported and the Rb concern remains a scoped limitation; if they do not, the strongest claim fails. Therefore the verdict stays CONDITIONAL, but the condition should explicitly include statistical validation of the reported improvements.","tokens_in":14173,"tokens_out":4396,"duration_ms":45704,"concrete_test":"Run a paired bootstrap or permutation test on the per-run outputs underlying Tables 3 and 4 (10 runs for ROUGE-2, 5 runs for relevance/NPV/accuracy) for each dataset, comparing PhAITV against TAM, LAM_LEX, and AITV. Report p-values and 95% confidence intervals for every comparison. If any headline comparison (e.g., PhAITV vs TAM on Gay Rights ROUGE-2, or PhAITV vs AITV on 4Forums accuracy) has p ≥ 0.05, the 'significant improvement' claim is not supported for that setting. If raw per-run scores are unavailable, rerun the pipeline with at least 10 seeds and report the full score distributions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is the abstract's assertion that the pipeline 'shows a significant improvement over state-of-the-art methods.' This is a statistical claim, but the evaluation provides no statistical support for it. Section 4.4 reports Table 3 as 'Averages of ROUGE Measures' over 10 generated summaries, and Table 4 as median values over 5 runs. No standard deviations, confidence intervals, or hypothesis tests are reported for any model comparison. The only mention of statistical significance in Section 4.4 refers to a preliminary test about stop-word removal, not to the comparisons that substantiate the headline. Because the central claim rests directly on these tables, the observed margins (e.g., ROUGE-2 F-measure 4.6 vs 2.1 for Abortion; relevance 0.93 vs 0.53) could be within run-to-run noise. If a paired test fails to reject the null for one of the headline comparisons, the paper's central assertion is not established. This concern is more load-bearing than the Rb=1 assumption, which is at least supported by dataset statistics (67–78% rebuttals) and explicitly scoped in the conclusion; the statistical claim has no analogous support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an unsupervised pipeline for extracting contrastive sentential reasons from online debate forums. The pipeline consists of a phrase-mining stage, a new Phrase Author Interaction Topic-Viewpoint (PhAITV) model with collapsed Gibbs sampling, a grouping and facet-labeling stage, and an extraction stage that produces a two-column digest table of reasons supporting opposing viewpoints. The authors evaluate the pipeline on CreateDebate, 4Forums, and Reddit datasets, comparing against TAM, LAM_LEX, and a unigram AITV variant. Evaluation covers ROUGE-based informativeness, human-judged relevance, and viewpoint clustering accuracy. The abstract claims that the pipeline 'shows a significant improvement over state-of-the-art methods,' but the reported evaluation does not include the statistical machinery needed to support that claim.","tokens_in":14467,"tokens_out":3630,"duration_ms":38853,"significance":"If the empirical claims held, this would be a useful contribution to unsupervised contrastive summarization and argument mining. The model is clearly specified, the pipeline is concrete and reproducible in structure, and the evaluation uses multiple datasets, external baselines, and human annotations. The phrase-level topic-viewpoint modeling and the use of author-reply interactions are plausible design choices. However, the strongest claim in the abstract is currently not supported by the evidence: no confidence intervals, variance measures, or hypothesis tests are reported for the headline comparisons, and model selection is performed on the evaluation data. These issues are load-bearing because the paper's contribution is explicitly framed as a significant empirical improvement. The Rb=1 assumption is a limitation that the authors acknowledge and that is partially supported by dataset statistics, so I view it as secondary to the missing statistical support.","major_comments":[{"comment":"The abstract's claim of 'significant improvement' is not supported by the reported statistics. Table 3 reports only averages over 10 runs without standard deviations or confidence intervals, and Table 4 reports medians over 5 runs without any measure of dispersion. No paired significance test is reported for any comparison between PhAITV and the baselines. This matters in practice: on Gay Rights, PhAITV's ROUGE-2 values (2.7 recall, 2.8 F-measure) are identical to AITV's, so the headline claim cannot be assessed from the numbers as presented. Please provide per-run results, confidence intervals, or paired tests (e.g., Wilcoxon signed-rank or bootstrap) for the comparisons underlying the abstract's claim.","section":"Section 4.4, Table 3 and Section 4.5, Table 4"},{"comment":"Model selection is performed on the same data used for evaluation. The authors report choosing K by measuring NPMI coherence on the Topic-Viewpoint clusters of words and selecting hyperparameters by testing combinations that give a 'satisfying overall performance' on the datasets. Since the baselines are run with default parameters while PhAITV is tuned on the evaluation data, the comparison may be biased in PhAITV's favor. Please use a held-out validation split or a nested cross-validation procedure for K and hyperparameter selection, or report a sensitivity analysis showing that the headline conclusions are stable across reasonable choices of K and the Dirichlet parameters.","section":"Section 4.2"},{"comment":"The rebuttal variable Rb is fixed to 1 for every reply, and the viewpoint sampling in Eq. (2) depends on this assumption. While the CreateDebate and 4Forums columns of Table 2 report rebuttal rates between 66% and 78%, no rebuttal rate is reported for the Reddit dataset, and the model is applied to Reddit under the same fixed-Rb assumption. Please report the rebuttal rate for the Reddit data or include a sensitivity analysis that relaxes the Rb=1 assumption, and state how the relevance and clustering results change under alternative settings.","section":"Section 3.2, Eq. (1) and Table 2"}],"minor_comments":[{"comment":"The abstract contains a typo ('conv eyed'), which should read 'conveyed'.","section":"Abstract"},{"comment":"The product indices in the Gibbs sampling equations are written as starting at j=0, but the text describes sampling words within a phrase of length M; please clarify the index bounds and define all count variables explicitly at first use.","section":"Section 3.2, Eqs. (3) and (4)"},{"comment":"The phrase 'grp.cumulatif NbAssign' mixes French and English and is not defined; it should be 'grp.cumulativeNbAssign' with a formal definition. Also, the containment step in lines 21-30 may transfer assignments from a phrase to multiple containing phrases; please specify how conflicts are resolved when more than one containing phrase is a Verbal Expression.","section":"Algorithm 1"},{"comment":"Report inter-annotator agreement for the argument-facet annotation task. The paper reports 'average Kappa agreement' of 0.66 only for the relevance task in Section 4.5, but no agreement measure is given for the two annotators in Section 4.3.","section":"Section 4.3"},{"comment":"The phrase 'notwithstanding PhAITV view' in the table note is unclear; please rephrase to state that PhAITVview is an oracle variant with true viewpoint labels, and that bold-face comparison excludes it.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the model is a reasonable contribution. The main risk is that the central empirical claim is not statistically supported as reported. If the authors can add paired significance tests and validation-based model selection, I would be inclined to accept; without those, the 'significant improvement' claim is not established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the paper's central idea is better than its abstract's statistical claim. PhAITV, which extends the authors' AITV from unigrams to mined phrases and adds a grouping-and-labeling layer plus a sentence-extraction step, is a genuine step forward in unsupervised contrastive summarization of online debates. The example digests in Table 5 look substantially more useful than the TAM and LAM_LEX outputs.\n\nWhat's genuinely good: the model is coherent, the Gibbs sampler equations are laid out, and the evaluation uses multiple datasets (CreateDebate, 4Forums, a scraped Reddit thread), human annotation for relevance and stance clustering, and a degenerate unigram baseline to isolate the phrase-modeling contribution. That is more than most papers in this area do.\n\nThe soft spot is exactly where the stress-test points: the phrase \"significant improvement\" appears in the abstract and the body, but there are no significance tests, confidence intervals, or variances anywhere in Tables 3 and 4. The paper reports averages and medians over 5-10 runs. A paired t-test or Wilcoxon test would be the obvious addition. The margins look large, so I'd bet the main comparisons survive, but the paper doesn't demonstrate it. That is a load-bearing gap because the headline claim is statistical.\n\nA second, minor issue: the number of topics K is chosen by NPMI coherence on the test data, and hyperparameters are tuned on the same data. That leaks test information and nudges results upward. It's common in this literature, but it weakens the \"unsupervised\" claim slightly.\n\nThe Rb=1 assumption — every reply to another author is a rebuttal — looks crude but is honestly scoped: the authors cite the 67-78% rebuttal rates in their own data and note in the conclusion that other platforms may differ. I'd call that a moderate limitation, not a fatal one.\n\nNo code or data are released, which makes the human evaluation hard to check. Again, not unusual for 2019, but it limits reproducibility.\n\nWho should read it: anyone in argument mining or opinion summarization who wants an unsupervised alternative to supervised pipelines. It deserves a serious referee, because the core model is sound and the evaluation gap is fixable with better reporting rather than a redesign. I'd accept it with major revisions, mainly calling for significance testing and held-out model selection.","headline":"The pipeline is a real advance, but the paper's 'significant improvement' claim rests on tables with no error bars or significance tests.","tokens_in":14955,"tokens_out":2338,"would_cite":true,"duration_ms":22851,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an unsupervised pipeline can extract and cluster the reasons given by opposing sides in online debates, producing a two-column contrastive digest table without stance labels.","keywords":["contrastive summarization","argument facet","topic-viewpoint model","online debate","phrase mining","unsupervised clustering","stance detection","rebuttal"],"falsifier":"Run the same pipeline on a debate corpus in which a manual rating shows that fewer than half of replies attack the previous post, then compare the inferred viewpoint clusters with the authors' known stances. If clustering accuracy and relevance rates fall to the level of the unigram or random baselines, the reply-as-rebuttal assumption is doing the work and the claimed advantage does not transfer beyond debate forums with high rebuttal rates.","tokens_in":13965,"feed_emoji":"🗣️","tokens_out":4839,"duration_ms":46926,"temperature":0.7,"pith_summary":"This paper tries to show that the reasons people give on opposing sides of an online debate can be found and arranged into a two-column digest table without any supervision. It claims that detecting argument facets as multiword phrases, rather than single words, and using who-replies-to-whom as a signal of disagreement, is enough to cluster extracted reasons by viewpoint. On CreateDebate, 4Forums, and Reddit data, the proposed pipeline outperforms two existing contrastive summarization methods (TAM and LAM_LEX) and its own unigram version in relevance, clustering accuracy, and summary informativeness. The payoff, if true, is a tool that turns raw forum threads into a concise pro-versus-con overview of an issue without stance labels or training data.","feed_headline":"Phrase model splits online debates into pro and con reasons","feed_subtitle":"A reply-aware topic model builds pro/con digest tables with no stance labels, outperforming two earlier methods.","key_machinery":"The load-bearing object is PhAITV, the Phrase Author Interaction Topic-Viewpoint model: a generative model in which an author first chooses a viewpoint, and each multiword phrase in a post is either drawn from a background distribution or assigned a topic and drawn from a topic-viewpoint word distribution. Reply structure enters through equation (1), which makes the probability that a reply is a rebuttal depend on whether the reply's viewpoint differs from the viewpoints of the parent author's posts; the model fixes the rebuttal indicator $\\mathrm{Rb}=1$ for all cross-author replies, so the viewpoint sampler is pushed to choose viewpoints opposed to a post's parents and children. This adversarial pressure, combined with topic-viewpoint assignment of phrases, is what lets the pipeline separate the reasons of the two sides without stance labels.","core_discovery":"The central discovery is that a topic-viewpoint model operating on phrases instead of unigrams, augmented with the assumption that a reply usually rebuts the author it answers, can jointly detect argument facets and assign posts to viewpoints. In the PhAITV model, each phrase is assigned a topic and a viewpoint; phrases assigned to the same topic-viewpoint cell are then grouped by word overlap into clusters treated as argument facets, and the most representative phrase becomes a facet label. A final extraction step picks a short sentence from the posts that realizes each label, filling the digest table. The paper reports that this phrase-based pipeline yields higher relevance rates and better viewpoint clustering than TAM and LAM_LEX on all tested datasets, and that its ROUGE-2 summary scores approach those of correct reference summaries on one dataset. The authors present the result as evidence that facet-guided phrase modeling plus reply interactions is a productive route to contrastive summarization of polarized text.","pith_inferences":["A natural extension is to replace the fixed rebuttal indicator with a learned or lexicon-based disagreement score, which could let the same model work on social media platforms where supportive or neutral replies are common.","The grouping stage could be retested with noun-phrase-aware or hyponym-based similarity instead of verb-expression preference, which might recover general facets like 'birth control' more consistently.","Because the paper omits a separate evaluation of post-level viewpoint assignments, a testable next step is to compare PhAITV's inferred post stances against known author stances on the 4Forums data to see whether the viewpoint clustering is accurate beyond the final digest.","The ROUGE comparison on CreateDebate is built from reference summaries that combine one sentence per annotated reason label, so a direct gold-standard evaluation on 4Forums would test whether the summary informativeness advantage generalizes beyond reason-annotated data."],"forward_implications":["A purely unsupervised pipeline can output pro/con digest tables with reasons relevant enough for human annotators to infer the stance of each sentence.","Multiword phrase modeling carries much of the gain: PhAITV beats its unigram counterpart AITV on relevance and summary informativeness, so facet detection benefits from phrase-level units.","Author reply interactions are a usable stance signal: even without true viewpoints, the model approaches the performance of PhAITV_view, where true viewpoints are given, especially on negative predictive value.","The pipeline extends to datasets without any reason or stance annotations: it produced digest tables for 4Forums and Reddit datasets where only text and reply structure are available."],"supporting_citations":[{"why":"Supplies the phrase-mining procedure that partitions posts into multiword phrases before modeling.","marker":"[6]"},{"why":"Provides the CreateDebate reason-annotated corpus and reason labels used to construct reference summaries.","marker":"[9]"},{"why":"Defines the TAM baseline and the contrastive summarization task the pipeline is compared against.","marker":"[16]"},{"why":"Defines the LAM_LEX baseline for perspective-based summarization that the pipeline outperforms.","marker":"[27]"},{"why":"Provides the unigram AITV model whose degradation isolates the contribution of phrase modeling and grouping.","marker":"[26]"},{"why":"Supplies the 4Forums Internet Argument Corpus used as an unlabeled testbed and for stance-label checks.","marker":"[1]"},{"why":"Supplies the notion of argument facet that the grouping and labeling stage is built around.","marker":"[12]"}],"fun_headline_variants":["Phrase model mines pro/con reasons from debate replies","Reply-aware topics extract contrastive debate reasons","Phrase-level model beats baselines at debate summarization","Reply-aware phrase clusters beat prior debate models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that almost every reply in a debate attacks the post it replies to, so it treats all cross-author replies as rebuttals; if a forum's replies are mostly supportive or neutral, the inferred viewpoints and the clustering built on them become unreliable.","fun_headline_variants_meta":{"raw":{"variants":["Phrase model mines pro/con reasons from debate replies","Reply-aware topics extract contrastive debate reasons","Phrase-level model beats baselines at debate summarization","Reply-aware phrase clusters beat prior debate models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000675,"raw_usage":{"total_tokens":2998,"prompt_tokens":801,"completion_tokens":2197,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":417,"completion_tokens_details":{"reasoning_tokens":2137}},"tokens_in":417,"tokens_out":2197,"duration_ms":15546,"temperature":1.0,"reasoning_tokens":2137,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:41:00.407287+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline on a debate corpus in which a manual rating shows that fewer than half of replies attack the previous post, then compare the inferred viewpoint clusters with the authors' known stances. If clustering accuracy and relevance rates fall to the level of the unigram or random baselines, the reply-as-rebuttal assumption is doing the work and the claimed advantage does not transfer beyond debate forums with high rebuttal rates.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the phrase-mining procedure that partitions posts into multiword phrases before modeling."},{"cited_title":"In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP )","cited_arxiv_id":null,"evidence_quote":"Provides the CreateDebate reason-annotated corpus and reason labels used to construct reference summaries."},{"cited_title":"In: Proceedings of the 2010 Conference on Empirical Me thods in Natural Language Processing","cited_arxiv_id":null,"evidence_quote":"Defines the TAM baseline and the contrastive summarization task the pipeline is compared against."},{"cited_title":"In: Proceedings of the 2017 Conference on Empirical Methods in Natural Langu age Processing","cited_arxiv_id":null,"evidence_quote":"Defines the LAM_LEX baseline for perspective-based summarization that the pipeline outperforms."},{"cited_title":"In: Procee dings of the AAAI Inter- national Conference on Web and Social Media (ICWSM)","cited_arxiv_id":null,"evidence_quote":"Provides the unigram AITV model whose degradation isolates the contribution of phrase modeling and grouping."},{"cited_title":"In: LREC (2016)","cited_arxiv_id":null,"evidence_quote":"Supplies the 4Forums Internet Argument Corpus used as an unlabeled testbed and for stance-label checks."},{"cited_title":"I n: Proceedings of the 2015 Conference of the North American Chapter of the Asso ciation for Computational Linguistics: Human Language Technologies","cited_arxiv_id":null,"evidence_quote":"Supplies the notion of argument facet that the grouping and labeling stage is built around."}],"review_version":1}