{"id":"ca062ce9-367d-4c1f-8a68-0966937c5ba7","arxiv_id":"2507.17409","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Storytelling and hedging help subjective persuasion in online debate but hurt objective argument quality, while emotions show mostly domain-independent effects.","lead":"This paper measures how storytelling, emotions, and hedging relate to argument strength in two datasets, one scoring objective quality and one scoring personal persuasion. It finds that these features can help one kind of argument while hurting the other, and it releases the new annotations and code.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central storytelling/hedging contrast could be an artifact of unvalidated cross-domain feature annotations; the IBM ARGQ storytelling coefficient rests on ~42 machine-predicted positives with no target-domain check.","rationale":"The paper's central empirical contribution is the contrast between IBM ARGQ (objective quality) and Cornell CMV (persuasion) regression coefficients for storytelling and hedging, plus the emotion patterns. All of these coefficients are estimated from automatically predicted independent variables. The authors' own Limitations paragraph acknowledges that the emotion annotations are trusted because of held-out training performance, and no gold annotations exist for the target corpora. This is the load-bearing point: if the predicted labels are noisy in a way that is correlated with the outcome (e.g., a storytelling classifier that confuses first-person opinionated text with narrative, or an emotion classifier that fires on narrative style), the estimated effects are biased, and the sign-reversal between corpora could be a measurement artifact rather than a real property of argument quality. The sparsity of storytelling predictions in IBM ARGQ (0.8%, ~42 positives) makes the negative coefficient particularly vulnerable: a handful of false positives among 42 instances can determine the sign. The alternative concerns (confounded dataset comparison, multiple comparisons, overclaimed emotion-rhetoric hypothesis) are real but secondary: the dataset confound is acknowledged by the authors, and the core contrast remains informative as a two-dataset finding; multiple comparisons are unlikely to change the highly significant storytelling/hedging results; and the 'rhetoric utilization' statement is clearly a post-hoc hypothesis in the Discussion. The proposed manual annotation check is feasible (a few hundred instances) and would directly test whether the regressions survive with gold IVs. If they do, the paper's conditional acceptance is justified; if not, the headline claim would need to be withdrawn or substantially reframed.","tokens_in":17037,"tokens_out":7720,"duration_ms":77648,"concrete_test":"Manually annotate a stratified random sample of target instances—e.g., 150 IBM ARGQ and 150 Cornell CMV comments, oversampling the predicted-positive storytelling and emotion cases and covering the full predicted-probability range—using the paper's feature definitions. Compute precision/recall and calibration of each automated annotation on this sample. Then re-run the individual regressions from Table 3 using the gold annotations (binary labels or averaged human probabilities) as independent variables. If the IBM ARGQ storytelling coefficient stays significantly negative and the Cornell CMV coefficients stay significantly positive with comparable magnitudes, the concern is resolved; if coefficients attenuate, flip, or lose significance, the headline contrast is an artifact of measurement error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Sec. 5 depends on treating the automatically predicted storytelling, emotion, and hedging annotations from Sec. 4 as valid measurements on IBM ARGQ and Cornell CMV. The only reported evidence for the classifiers is held-out performance on their training domains (Table 5); no evaluation is run on the target corpora. This matters because the two target domains are far from the training data: the emotion classifiers are trained on CROWD-ENVENT event descriptions, and the storytelling classifier on mixed narrative/forum texts. The headline contrast is especially fragile for IBM ARGQ storytelling, where only 0.8% of instances are predicted positive (about 42 of 5.3k, Table 2), so the negative coefficient (-0.182) is identified by a very small set of out-of-domain predictions that could be false positives. Differential misclassification across the two corpora—for example, if the emotion or storytelling models systematically fire on the longer, more narrative Reddit posts but not on the short impersonal debate arguments—could create or inflate the opposite-signed coefficients. The Limitations paragraph explicitly relies on the held-out training data rather than any target-domain verification. Without a target-domain gold check, the regression coefficients and the claimed contrast are not secured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies the joint impact of three subjective features—storytelling, emotions, and hedging—on two argument-strength corpora. The authors automatically annotate IBM ARGQ (5.3k short debate arguments, objective quality scores) and Cornell CMV (11.5k Reddit comments, binary persuasion labels) using a RoBERTa ensemble for storytelling and ten emotion classifiers trained on CROWD-ENVENT, plus lexicon-based hedge detection. They then run univariate and stepwise regressions with interactions. They report that storytelling and hedging have negative coefficients on IBM ARGQ but positive effects on Cornell CMV persuasiveness, while discrete emotions have largely consistent effects across corpora, with disgust and guilt/shame negative and fear/sadness positive. The paper also contributes the annotated datasets and code.","tokens_in":17253,"tokens_out":7183,"duration_ms":80838,"significance":"If the empirical results hold, the main contrast—subjective features helping persuasion but hurting objective quality—would be a valuable contribution to computational argumentation, and the released annotated datasets would support further research. The paper has genuine strengths: it compares multiple annotation strategies (e.g., masked versus unmasked emotion training, mixed-domain ensembles), it reproduces prior storytelling results, and it is unusually explicit about the limitations of its cross-domain annotation. The data and code are shared. However, the significance of the empirical findings is currently limited by the absence of target-domain validation of the automatically predicted features and by inferential choices discussed below.","major_comments":[{"comment":"The independent variables for storytelling and emotions are machine predictions with no target-domain validation. For IBM ARGQ, the storytelling classifier predicts only 0.8% positives (45 of 5.3k instances in Table 2), and all reported held-out F1 scores in Table 5 come from training-domain test sets (Falk and Lapesa 2022; CROWD-ENVENT), not from IBM ARGQ or Cornell CMV. The Limitations section explicitly justifies the emotion annotations via good results on the heldout training data. Measurement error in binary predictors is non-classical and can bias regression coefficients in either direction, especially with rare positives. Since the headline storytelling/hedging contrast is identified by these predicted labels, the authors should supply target-domain validation (for example, manual evaluation of a random sample, particularly the 45 IBM ARGQ storytelling positives), run a sensitivity analysis under misclassification assumptions, or substantially soften the contrast claims.","section":"§4.1, §4.2, Table 2, Limitations"},{"comment":"The central contrast between objective argument quality and individualized persuasion is confounded by domain differences. IBM ARGQ and Cornell CMV differ simultaneously in text length, register, collection procedure, annotation process, and topic, as the authors acknowledge in §3 when they state that the number of differences disallows a comparison without confounding factors. The opposite signs for storytelling and hedging may therefore reflect domain, length, or register effects rather than the objective/subjective distinction. The paper should frame the result as a comparison of two datasets rather than of two argument-strength constructs, or add within-domain controls such as length, topic, or matched sub-samples.","section":"§3, Discussion"},{"comment":"The stepwise regression procedure selects features and interactions by AIC and significance tests on the same data, and the resulting p-values are reported without correction for this selection. In addition, Table 3 tests multiple emotion categories plus storytelling and hedging variants against two corpora, and some Cornell CMV effects are only marginal (for example storytelling p=0.015, pride p=0.042, relief p=0.007) with no multiple-comparison correction. With final explained variance of 3.96% for IBM ARGQ and 1.36% for Cornell CMV, the Limitations statement that high significance nonetheless shows importance overstates the evidence, because post-selection p-values are not valid and the effect sizes are very small. The authors should hold out a validation set for model selection, correct for multiple testing, and report effect sizes with confidence intervals.","section":"§5, Table 9"},{"comment":"The abstract claim that the influence of emotions depends on their rhetoric utilization rather than the domain is not supported by the regression design. The models measure emotion categories such as anger, fear, and disgust, but not rhetoric utilization such as emotional attack versus appeal; the attack/appeal distinction is introduced post hoc from a few examples in Table 4 and a qualitative reading. To support this claim, the authors would need to annotate or automatically measure attack/appeal usage and include it in the regression, or explicitly label the claim as an untested hypothesis.","section":"§5 Discussion, Conclusion"}],"minor_comments":[{"comment":"The sentence listing the features as 1 storytelling plus 9 emotions excluding boredom and surprise plus 6 hedging variants does not match Table 3, which includes boredom and reports only one hedge count; please harmonize the feature count and the table.","section":"§5"},{"comment":"The expression beta-story = 0.138, beta-hedge) = 0.030 contains a misplaced parenthesis; please clarify whether the Cornell CMV estimates are log-odds or odds ratios.","section":"§5"},{"comment":"The header standard deviance should be standard deviation; the current wording is ambiguous and nonstandard.","section":"Table 5"},{"comment":"The storytelling training data includes a subset of r/ChangeMyView, so the Cornell CMV predictions are not fully cross-domain; please state this explicitly when discussing cross-domain robustness.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent about its limitations, but the central empirical result depends on unvalidated feature predictions and on post-selection inference. I would ask for target-domain validation of the storytelling and emotion annotations, a more careful treatment of multiple comparisons and stepwise selection, and a softening of the rhetoric-utilization claim. The reuse of Falk and Lapesa (2022) data is legitimate but should be disclosed more prominently in the contribution framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this paper has a real new result—the first joint regression of storytelling, emotion, and hedging against both objective quality and subjective persuasion—but the headline contrast is only as solid as the automatically predicted features, and those are not validated on the target corpora. I’d send it to review, but with a request for validation before acceptance.\n\nWhat’s genuinely new: the joint modeling. Prior work studied these features in isolation, so the opposite-signed effects for storytelling and hedging across IBM ARGQ and Cornell CMV are a useful empirical contribution. The released annotated datasets will be useful to the community. The emotion findings—mostly stable directions, with guilt/shame and disgust negative and fear/sadness positive—are plausible and align with earlier isolated results.\n\nThe soft spots are real and concentrate in the annotation pipeline. The classifiers are evaluated only on held-out training data, not on the actual target texts. That matters because the training domains are far from the targets: CROWD-ENVENT event descriptions for emotions, and mixed narrative/forum texts for storytelling. The IBM ARGQ storytelling cell is tiny—about 42 predicted positives out of 5.3k—so the negative coefficient is identified by very few out-of-domain predictions. If the classifier fires differently on short impersonal debate arguments versus long Reddit posts, the opposite signs could be an artifact of differential misclassification. The stress-test note is right. The paper’s own Limitations paragraph explicitly relies on heldout performance, which is not enough.\n\nThere are also smaller issues: stepwise model selection with no multiple-comparison correction, very low explained variance (3.96% r2 on IBM ARGQ, 1.36% pseudo-r2 on Cornell), and the abstract states the 'rhetoric utilization' interpretation as a finding when it’s only a hypothesis from a few examples. The self-citation to Falk and Lapesa is minor and appropriate—they’re replicating that setup.\n\nBottom line: the central contrast is plausible and consistent with prior isolated findings, but it is not secured. A reviewer should ask for a sample of target-domain annotations (even a few hundred per corpus), an overlap check between the storytelling training set and Cornell CMV, and a sensitivity analysis that re-runs the regressions on the subset of instances with high-confidence predictions.\n\nFor a workshop paper, this is the right kind of ambition. It deserves serious refereeing, but conditional on the validation. I’d bring it to reading group mostly to debate how much weight to put on cross-domain annotation.","headline":"A genuinely new joint regression with a plausible but unsecured cross-domain annotation pipeline; worth refereeing with validation requested.","tokens_in":17788,"tokens_out":2332,"would_cite":false,"duration_ms":25255,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Storytelling and hedging boost subjective persuasion but suppress objective argument quality in regression tests on two datasets.","keywords":["argument strength","persuasion","argument quality","storytelling","emotions","hedging","regression analysis","automated annotation"],"falsifier":"A manual annotation of a random sample of 200–300 arguments from each dataset by human judges for storytelling, the ten emotions, and hedging, followed by re-running the regressions with the human labels, would settle whether the automated-label coefficients reflect real effects or measurement error; if the signs or significances change, the paper's central contrast is an artifact of the annotation pipeline.","tokens_in":16816,"feed_emoji":"📊","tokens_out":10524,"duration_ms":86873,"temperature":0.7,"pith_summary":"This paper investigates whether three subjective rhetorical devices—personal storytelling, emotional language, and hedging (uncertainty markers like 'probably' or 'I think')—strengthen or weaken arguments, and whether their impact depends on how argument strength is defined. Using regression on two standard datasets, IBM ARGQ for objective argument quality and Cornell CMV for subjective persuasion, the authors find that storytelling and hedging significantly help persuasion but significantly hurt objective quality. Emotion effects are mostly stable across the two datasets: guilt/shame and disgust reduce argument strength, while fear and sadness increase it, and the authors interpret this as evidence that the rhetoric use of emotions matters more than the domain. The paper also contributes automated annotation methods for the three features, since no existing dataset labels all of them.","feed_headline":"Storytelling and hedges help persuade but hurt objective quality","feed_subtitle":"Regression on two argument datasets shows opposite effects for storytelling and hedging, with emotions stable across domains.","key_machinery":"The central object is a pair of parallel regressions on two datasets that operationalize objective and subjective argument strength: OLS linear regression on IBM ARGQ's continuous quality score and logistic regression on Cornell CMV's binary persuasion label. The independent variables come from automated annotation layers: a 10-fold RoBERTa ensemble for storytelling trained on mixed domains, masked emotion classifiers per emotion trained on CROWD-ENVENT event descriptions, and a lexicon-based hedge detector with syntactic disambiguation rules. The design's power comes from the contrast between the datasets, which differ in collection, length, and annotation procedure, so any feature whose coefficient sign is consistent across both is attributed to the feature itself, while a sign flip is attributed to the difference between objective and subjective strength.","core_discovery":"The central claim is that the value of a subjective rhetorical feature is not intrinsic; it flips sign with the facet of argument strength being measured. On IBM ARGQ, where short impersonal arguments are scored by averaged crowd judgments of adequacy, storytelling has a significant negative coefficient ($\\beta = -0.182$) and hedge count is significantly negative ($\\beta = -0.011$). On Cornell CMV, where a delta means one reader changed their mind, storytelling raises the odds of persuasion by a factor of 1.148 and hedge count by 1.030. Emotions behave differently: disgust and guilt/shame significantly lower argument strength in both datasets, while fear and sadness significantly raise it, which the authors trace to the difference between emotional attacks aimed at a participant and emotional appeals to universal concerns. The regression models themselves explain only a few percent of variance, so the contribution is about the direction and significance of the effects, not their size.","pith_inferences":["A direct test of the paper's interpretation would be to re-run the analysis with a third dataset that contains both objective and subjective strength labels for the same arguments; the framework predicts that feature coefficients align with the label definition, not with the topic or platform.","The emotion results suggest that classifying emotional attacks versus emotional appeals, rather than raw emotion categories, could be a stronger predictor of argument strength than any single emotion label.","The automated annotation transfer assumption is fragile: because classifiers are validated only on training-domain held-out data, a manual validation sample from the target corpora would reveal whether the sign flips are genuine or measurement artifacts."],"forward_implications":["Argument-quality models built for objective settings such as essays or debates should treat personal stories and hedges as likely weaknesses, not strengths.","Persuasion-oriented systems, in contrast, can treat hedges and personal narratives as positive signals of credibility-building.","Emotion-aware argument models can expect the same directional effects across domains: disgust and guilt/shame lower strength, fear and sadness raise it.","Because explained variance is low (about 4% for quality and 1% for persuasion), these features complement rather than replace contextual and demographic predictors."],"supporting_citations":[{"why":"Supplies the IBM ARGQ dataset with objective argument quality scores used as the dependent variable.","marker":"Toledo et al., 2019"},{"why":"Supplies the Cornell CMV dataset with delta-based persuasion labels used as the dependent variable.","marker":"Tan et al., 2016"},{"why":"Provides the storytelling training data and the mixed-domain ensemble setup replicated for annotation.","marker":"Falk and Lapesa, 2022"},{"why":"Supplies the CROWD-ENVENT emotion event descriptions used to train the per-emotion classifiers.","marker":"Troiano et al., 2019"},{"why":"Provides the hedging lexicon and disambiguation rules used by the rule-based annotator.","marker":"Islam et al., 2020"},{"why":"Justifies the deep-ensemble approach for stable predictions and interpreting average probabilities as certainty.","marker":"Lakshminarayanan et al., 2017"}],"fun_headline_variants":["Storytelling persuades but fails objective tests","Subjective rhetoric flips sign across argument datasets","Hedging: persuasive but not 'good' argument quality","Emotions in argument: context decides their weight","Persuasion vs. quality: storytelling splits the difference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automated annotations of storytelling, emotions, and hedges are accurate enough on the target corpora to serve as independent variables, but they are only validated on held-out training-domain data, not on IBM ARGQ or Cornell CMV text.","fun_headline_variants_meta":{"raw":{"variants":["Storytelling persuades but fails objective tests","Subjective rhetoric flips sign across argument datasets","Hedging: persuasive but not 'good' argument quality","Emotions in argument: context decides their weight","Persuasion vs. quality: storytelling splits the difference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1252,"prompt_tokens":914,"completion_tokens":338,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":263}},"tokens_in":530,"tokens_out":338,"duration_ms":4370,"temperature":1.0,"reasoning_tokens":263,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:47:46.739178+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A manual annotation of a random sample of 200–300 arguments from each dataset by human judges for storytelling, the ten emotions, and hedging, followed by re-running the regressions with the human labels, would settle whether the automated-label coefficients reflect real effects or measurement error; if the signs or significances change, the paper's central contrast is an artifact of the annotation pipeline.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the IBM ARGQ dataset with objective argument quality scores used as the dependent variable."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the storytelling training data and the mixed-domain ensemble setup replicated for annotation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the hedging lexicon and disambiguation rules used by the rule-based annotator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Justifies the deep-ensemble approach for stable predictions and interpreting average probabilities as certainty."}],"review_version":1}