Pith. sign in

REVIEW 3 major objections 5 minor 30 references

A Dataset of General-Purpose Rebuttal

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 55-entry knowledge base of general claims and their counterarguments can rebut speeches across many debate topics, with 87% of pre-written rebuttals judged plausible.

desk verdict A genuinely useful dataset paper; the 87% rebuttal-plausibility result is thinner than the abstract suggests, but the dataset itself justifies a serious look. read the letter →

arxiv 1909.00393 v1 pith:VFPRLTVV submitted 2019-09-01 cs.CL cs.AIcs.LG

Matan Orbach , Yonatan Bilu , Ariel Gera , Yoav Kantor , Lena Dankin , Tamar Lavee , Lili Kotlerman , Shachar Mirkin
show 3 more authors
Michal Jacovi Ranit Aharonov Noam Slonim
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords general-purposerebuttalgenerationargumentminingdebatedatasetlisteningcomprehensionclaimdetectioncrowdsourcedannotationknowledgebase
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a new task: automatically producing a critical rebuttal to a long argumentative speech, rather than to a short utterance. It argues that a small set of general-purpose claims, statements that recur across many debate topics, can be paired with pre-written counterarguments and reused to rebut speeches on topics never seen in advance. To support this, the authors built the GPR-KB-55 dataset: 55 manually authored claim-rebuttal pairs with annotations linking them to 50 debate motions and 200 recorded speeches. The annotations show that 46% of claim-motion pairs are relevant, that at least one general claim is mentioned in every speech, and that 87% of the pre-written rebuttals were judged plausible in context. If correct, this gives automatic rebuttal systems a topic-independent starting point, and the dataset is released for research.

What carries the argument

The carrying object is the General-Purpose Rebuttal Knowledge Base (GPR-KB): 55 manually authored claim-rebuttal pairs in which each claim is a short sentence with slots such as '[ACTION] [TOPIC]' that get instantiated for a specific motion, and each rebuttal is a context-free counterargument. It carries the argument by being small enough to curate by hand yet general enough that annotators label 46% of claim-motion pairs as relevant, and it is paired with a three-stage annotation pipeline that measures relevance, stance, mention in speech, sentence-level location, and rebuttal plausibility.

What would settle it

Re-annotate the 3,246 speech-claim pairs under an adjudicated protocol with expert arbiters; if the share of speeches containing at least one GP-claim falls well below the paper's 100 percent coverage, the claim that the 55 general claims appear across all speeches would be refuted. Likewise, ask annotators to judge the same 103 speech-rebuttal pairs with rebuttals randomly swapped across claims: if the swapped rebuttals are judged plausible as often as the true ones, the 87 percent result would reflect generic acceptability rather than genuine counter-argument quality.

Watch

Extended reading notes

Core claim

The central discovery is that general-purpose argumentative claims are both frequent in real spoken debates and rebuttable without topic-specific knowledge. In the authors' terms, Section 4.2 shows GP-claims 'are often used in spoken content discussing various topics': 41% of relevant claim-speech pairs were labeled as mentioned (34% implicitly), and the 55 claims cover all 200 speeches, compared with 86.5% coverage for the topic-specific claims in the prior dataset. Section 4.4 shows that 'most rebuttals are appropriate in the vast majority of contexts': annotators judged 87% of 103 speech-rebuttal pairs plausible. A surprising baseline result is that simply predicting the most frequently mentioned claims, without reading the speech, competes with or beats text-based detectors, which the authors take as evidence that claim-frequency priors must be controlled for in evaluation.

Load-bearing premise

The load-bearing premise is that crowdsourced majority votes for whether a speaker mentioned a general claim are accurate enough to measure coverage, even though annotators rarely agree with each other (a standard agreement score of 0.37), and the rebuttal-plausibility result was collected without a control condition.

Editorial extensions

If this is right

  • A concise, hand-curated set of 55 general claims can supply at least one candidate rebuttal for every speech in the dataset, removing the need for topic-specific argument lists.
  • Pre-written, context-free rebuttals can be judged plausible responses in 87% of the cases tested, so automatic rebuttal systems can be built without generating fresh content per speech.
  • The prior baseline, predicting the claims most frequently mentioned in training speeches, is strong enough that any text-based claim detector must be compared against it, and outperforming it requires more than semantic similarity.
  • The GPR-KB-55 dataset roughly triples the average number of claim mentions found per speech relative to the topic-specific annotations (6.7 vs. 1.8), giving listening-comprehension research a larger test bed.
  • Relevance and stance of general claims to a motion can be annotated with moderate agreement, making it feasible to filter claims for a given topic before rebuttal selection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 87% plausibility rate may overestimate quality because annotators first decided the claim was mentioned and then judged its rebuttal, an anchoring setup; a control condition presenting the same rebuttals for claims they were not written for would test whether the rate reflects generic acceptability.
  • The low mention-detection agreement suggests 'implicit mention' is partly in the ear of the listener; an automatic system trained on these labels may be learning to reproduce what audiences read into a speech rather than what the speaker literally said, which may still be the right target for rebuttal.
  • Because only 5% of the 55 claims were relevant to all 50 motions, a deployed system would likely need to extend the GPR-KB or instantiate claims dynamically to keep coverage high on niche topics.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a new task in natural language understanding: producing a rebuttal in response to a long argumentative text, and proposes a General-Purpose Rebuttal Knowledge Base (GPR-KB) containing 55 manually authored general claims with matching rebuttals. The GPR-KB is evaluated through four annotation experiments over 200 speeches on 50 debate motions: cross-topic relevancy of claims, mention detection in speeches, sentence-level localization, and plausibility of rebuttals in context. The authors also provide baseline results for detecting whether a GP-claim is mentioned in a speech, showing that a frequency-based prior is a strong baseline. The central claims are that GP-claims are relevant across many topics, are commonly mentioned in spoken content, and that the pre-written rebuttals are judged plausible in 87% of 103 evaluated speech-rebuttal pairs.

Significance. If the findings are robust, the paper makes a useful contribution: it defines a novel task, releases a multi-layer dataset (GPR-KB-55), and demonstrates that a compact set of general claims can often be matched to real argumentative speeches. Strengths include the expert-debater authorship of the knowledge base without access to the target motions, the use of multiple annotation layers with known-answer control questions, and the honest reporting of a strong frequency-prior baseline. The paper also provides a clear comparison with the iDebate18 resource. However, the headline rebuttal-plausibility result currently rests on a single 87% majority-vote statistic that lacks a control condition and a stable denominator, and the mention-detection labels that drive coverage and sampling have only fair pairwise agreement. These issues are load-bearing for the paper's central empirical claims and require additional analysis or experiments.

major comments (3)
  1. [Section 4.4, Results] The central claim that 'most rebuttals are appropriate in the vast majority of contexts' rests on the 87% plausibility rate over 103 speech-rebuttal pairs, but the experiment has no control condition: annotators are shown the GPR-KB rebuttal only after marking the claim as mentioned, and are never offered generic rebuttals, rebuttals matched to a different claim, or non-rebuttal texts. Without such a control, the high 'plausible' rate may reflect acquiescence or a low decision bar rather than the specific content of the GPR-KB. Additionally, because only annotators who marked the claim as mentioned proceed to rate the rebuttal, the number of raters contributing to each majority label is variable and self-selected; the paper does not report the distribution of the number of ratings per pair. Please add a control condition, report the per-pair rater counts, and provide confidence intervals for the 87% estimate.
  2. [Section 4.2] The mention annotations used to sample the 103 rebuttal pairs and to compute coverage and prior statistics have pairwise Cohen's kappa of only 0.37, conventionally 'fair' agreement. The paper treats the majority label as ground truth throughout, but provides no reliability analysis of these majority labels beyond the 7% error rate on known-answer questions. If a substantial fraction of the sampled 'mentions' are false positives, the rebuttal-plausibility judgments in Section 4.4 are partially evaluated against claims the speaker never made. Please report agreement between individual annotators and the majority label (as is done for the rebuttal task), and perform a sensitivity analysis of the coverage, average-mention, and prior statistics under stricter mention thresholds (e.g., requiring 7 or more of 10 annotators, or unanimous agreement).
  3. [Section 4.4, Analysis] The manual analysis of unanimously inappropriate rebuttals is reported only qualitatively: 'this stemmed from the rebuttal being inappropriate for the topic, rather than a specific speech' is stated without counts, a coding scheme, or a quantitative follow-up. Given that the paper decides to stop collecting further annotation based in part on this analysis, please provide the number of unanimously negative cases, examples, and a systematic check of whether excluding or down-weighting topic-inappropriate cases changes the 87% estimate.
minor comments (5)
  1. [Section 4.4, first paragraph] The phrase 'would consist a plausible rebuttal response' should be 'would constitute a plausible rebuttal response' or 'would be a plausible rebuttal response.'
  2. [Section 5, last paragraph] The text uses 'F 1-score' with an awkward space; use 'F1 score' consistently throughout.
  3. [Section 4.3] The similarity thresholds for sentence-claim pairing differ (0.5 for GP-claims and 0.7 for iDebate claims) without explanation; a brief justification would help readers interpret the two annotation pools.
  4. [Table 3] The column header 'Annotated pairs' is redundant with the table title; consider renaming it 'Total pairs' for clarity.
  5. [Section 3.2] The sentence 'She was not given access to any of the iDebate18 motions' uses a gendered pronoun where 'the debater' or 'the author' would be more neutral, though this is a minor style point.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical claims are evaluated by independent crowd annotation and the self-citations are data provenance, not load-bearing reductions.

full rationale

The paper does not present a formal derivation whose conclusion is equivalent to its inputs; it reports a dataset and empirical measurements. The GPR-KB was authored by a debater with no access to the iDebate18 motions (§3.2), and the coverage, mention, and rebuttal-plausibility statistics come from crowd annotations over held-out speech-claim pairs. The prior baseline in §5 is explicitly a training-frequency estimator ('a-priori probability that a GP-claim will be mentioned... can be computed' and applied under leave-one-motion-out), so calling it a 'challenging baseline' is an honest benchmark, not a renamed prediction. Self-citations to Mirkin et al. (2018) identify the source of the 200 speeches and the Listening Comprehension task; they are provenance and comparison, and no uniqueness or first-principles claim is imported from those self-citations. The absence of a control condition for the 87% rebuttal-plausibility figure is an experimental-design weakness, not a circular reduction, and therefore does not raise the circularity score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The core experimental claims depend mainly on annotation thresholds, annotator quality choices, and the implicit assumption that crowd labels are trustworthy. No new theoretical entities are introduced. The GPR-KB is a dataset, not a postulated mechanism.

free parameters (4)
  • Relevance threshold for potential claim lists = at least 2 of 5 annotators (vs majority 3 in Section 4.1)
    In Section 4.2, claims with at least 2 annotations of relevance were included in the speech annotation, expanding the potential claim set and directly affecting coverage and mention statistics.
  • Claim-sentence similarity thresholds = 0.5 for GP-claims, 0.7 for iDebate claims
    In Section 4.3, only sentence-claim pairs with word2vec similarity above these thresholds were annotated, which shapes the positive rate and the training data for BERT.
  • Annotator quality control thresholds = at least 10 responses, at least 5 common answers with 3 peers, average peer agreement at least 0.2
    In Section 4.3, annotators failing these criteria were removed, affecting the final labels and agreement scores.
  • Claim prior probabilities = empirical mention frequencies from training motions
    The prior baseline in Section 5 is computed from training data and is applied to held-out motions. These values are fitted to data, but they are presented as a baseline rather than as the paper's predictive claim.
assumptions (4)
  • domain assumption Majority annotation approximates ground truth for relevance and mention
    All coverage and prior statistics treat majority-vote labels (or unanimous validation items) as correct, despite agreement scores ranging from fair to moderate.
  • domain assumption A single expert debater's authored claims are a representative sample of general arguments
    The GPR-KB was authored by one debater without access to the target motions (Section 3.2). The paper assumes her selection captures common general claims.
  • domain assumption The selected crowd annotators provide reliable labels
    The paper uses 22 preselected Figure Eight annotators and filters by quality criteria. It assumes their judgments are reliable despite a kappa of 0.37 for mention detection.
  • domain assumption Train/test motion splits generalize
    The prior and BERT baselines use leave-one-motion-out or random 80/20 motion splits. The paper assumes claim mention priors transfer across topics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Dataset of General-Purpose Rebuttal." pith.science (2026). https://pith.science/paper/VFPRLTVV

@misc{pith2026190900393,
  author       = {Pith},
  title        = {Pith review of: A Dataset of General-Purpose Rebuttal},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VFPRLTVV}},
  note         = {Machine review of arXiv:1909.00393}
}
read the original abstract

In Natural Language Understanding, the task of response generation is usually focused on responses to short texts, such as tweets or a turn in a dialog. Here we present a novel task of producing a critical response to a long argumentative text, and suggest a method based on general rebuttal arguments to address it. We do this in the context of the recently-suggested task of listening comprehension over argumentative content: given a speech on some specified topic, and a list of relevant arguments, the goal is to determine which of the arguments appear in the speech. The general rebuttals we describe here (written in English) overcome the need for topic-specific arguments to be provided, by proving to be applicable for a large set of topics. This allows creating responses beyond the scope of topics for which specific arguments are available. All data collected during this work is freely available for research.

Figures

Figures reproduced from arXiv: 1909.00393 by the authors.

Figure 1
Figure 1. Annotation overview: All motion-claim pairs [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The distribution of GP-claims vs. the number [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The distribution of GP-claims vs. speech [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Precision-Recall curves for the matching of [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Precision-Recall curves for matching GP [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 19 canonical work pages

  1. [1]

    Yonatan Bilu, Ariel Gera, Daniel Hershcovich, Benjamin Sznajder, Dan Lahav, Guy Moshkowich, Anael Malet, Assaf Gavron, and Noam Slonim. 2019. https://www.aclweb.org/anthology/P19-1097 Argument invention from first principles . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1013--1026, Florence, Italy. Ass...

    work page 2019

  2. [2]

    Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems: Recent advances and new frontiers. ACM SIGKDD Explorations Newsletter, 19(2):25--35

    work page 2017

  3. [3]

    Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales . Educational and Psychological Measurement, 20(1):37--46

    work page 1960

  4. [4]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    arXiv 2018

  5. [5]

    Phan Minh Dung. 1995. On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. Artificial intelligence, 77(2):321--357

    work page 1995

  6. [6]

    Xinyu Hua, Zhe Hu, and Lu Wang. 2019. https://www.aclweb.org/anthology/P19-1255 Argument generation with retrieval, planning, and realization . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2661--2672, Florence, Italy. Association for Computational Linguistics

    2019

  7. [7]

    Xinyu Hua and Lu Wang. 2018. https://doi.org/10.18653/v1/P18-1021 Neural argument generation augmented with externally retrieved evidence . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 219--230, Melbourne, Australia. Association for Computational Linguistics

    doi:10.18653/v1/p18-1021 2018

  8. [8]

    L \'a szl \'o A Jeni, Jeffrey F Cohn, and Fernando De La Torre. 2013. Facing imbalanced data--recommendations for the use of performance metrics. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, pages 245--251. IEEE

    work page 2013

Show all 30 references
  1. [9]

    Tamar Lavee, Matan Orbach, Lili Kotlerman, Yoav Kantor, Shai Gretz, Lena Dankin, Shachar Mirkin, Michal Jacovi, Yonatan Bilu, Ranit Aharonov, and Noam Slonim. 2019. http://arxiv.org/abs/1907.11889 Towards effective rebuttal: Listening comprehension using corpus-wide claim mini...

    2019 arXiv

  2. [10]

    Pierre - Emmanuel Mazar \' e , Samuel Humeau, Martin Raison, and Antoine Bordes. 2018. http://arxiv.org/abs/1809.01984 Training millions of personalized dialogue agents . In EMNLP

    2018 arXiv

  3. [11]

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. http://arxiv.org/abs/1301.3781 Efficient estimation of word representations in vector space . CoRR, abs/1301.3781

    2013 arXiv

  4. [12]

    Shachar Mirkin, Guy Moshkowich, Matan Orbach, Lili Kotlerman, Yoav Kantor, Tamar Lavee, Michal Jacovi, Yonatan Bilu, Ranit Aharonov, and Noam Slonim. 2018. http://aclweb.org/anthology/D18-1078 Listening comprehension over argumentative content . In Proceedings of the 2018 Conf...

    2018

  5. [13]

    Elena Musi, Debanjan Ghosh, Mark Aakhus, Smaranda Muresan, and Nina Wacholder. 2017. Building an ontology of (dis) agreement space for argument mining. SIGDIAL/SEMDIAL 2017 Joint Session on Negotiation Dialog (position paper)

    2017

  6. [14]

    Andreas Peldszus and Manfred Stede. 2015 a . An annotated corpus of argumentative microtexts. In Proceedings of the First Conference on Argumentation, Lisbon, Portugal, June. to appear

    2015

  7. [15]

    Andreas Peldszus and Manfred Stede. 2015 b . Towards detecting counter-considerations in text. In Proceedings of the 2nd Workshop on Argumentation Mining, pages 104--109

    2015

  8. [16]

    Alan Ritter, Colin Cherry, and William B Dolan. 2011. Data-driven response generation in social media. In Proceedings of the conference on empirical methods in natural language processing, pages 583--593. Association for Computational Linguistics

    2011

  9. [17]

    Sara Rosenthal and Kathy McKeown. 2015. I couldn’t agree more: The role of conversational structure in agreement and disagreement detection in online discussions. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 168--177

    2015

  10. [18]

    Andrew Shin, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2015. Context-dependent automatic response generation using statistical machine translation techniques. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Lin...

    2015

  11. [19]

    Dhanya Sridhar, James Foulds, Bert Huang, Lise Getoor, and Marilyn Walker. 2015. Joint models of disagreement and stance in online debate. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on N...

    2015

  12. [20]

    Hiroaki Sugiyama, Toyomi Meguro, Ryuichiro Higashinaka, and Yasuhiro Minami. 2013. Open-domain utterance generation for conversational dialogue systems using web-scale dependency structures. In Proceedings of the SIGDIAL 2013 Conference, pages 334--338

    2013

  13. [21]

    Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. https://doi.org/10.18653/v1/P18-1023 Retrieval of the best counterargument without prior topic knowledge . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)...

    2018 doi

  14. [22]

    Marilyn A Walker, Jean E Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012. A corpus for research on deliberation and debate. In LREC, pages 812--817. Istanbul

    2012

  15. [23]

    Joseph Weizenbaum. 1966. https://doi.org/10.1145/365153.365168 Eliza--a computer program for the study of natural language communication between man and machine . Commun. ACM, 9(1):36--45

    1966

  16. [24]

    Tsung - Hsien Wen, Milica Gasic, Nikola Mrksic, Lina Maria Rojas - Barahona, Pei - Hao Su, Stefan Ultes, David Vandyke, and Steve J. Young. 2016. http://arxiv.org/abs/1604.04562 A network-based end-to-end trainable task-oriented dialogue system . In EACL

    2016 arXiv

  17. [25]

    Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2017. Topic aware neural response generation. In Thirty-First AAAI Conference on Artificial Intelligence

    2017

  18. [26]

    Rui Yan, Yiping Song, and Hua Wu. 2016. Learning to respond with deep neural networks for retrieval-based human-computer conversation system. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 55--64. ACM

    2016

  19. [27]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  20. [28]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  21. [29]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence a...

  22. [30]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.