{"id":"0958f41a-9249-4e24-8463-cf1483eeb46f","arxiv_id":"1908.06083","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Iterative human adversarial attacks and retraining make offensive-language detectors in dialogue harder to fool, and dialogue context is critical for detecting those attacks.","lead":"Researchers trained offensive-language detectors by inviting crowdworkers to try to trick the current model, then retraining the model on the tricks that worked. Repeating this loop made detectors substantially harder to fool and showed that whether a reply is offensive often depends on the conversation around it.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 6's robustness gains are measured on same-round holdouts drawn from attacks aimed at the exact deployed predecessor; a truly fresh, held-out attack round is required to support deployment robustness.","rationale":"The reader's weakest_assumption identifies the same load-bearing issue: the test sets are generated against the exact model being evaluated, so cross-round improvements and zero scores are partly definitional. My reading agrees with that assessment. The paper's internal evidence is credible: the multi-turn experiments use five runs with standard deviations, the WTC comparisons are transparent, and the code/data release is promised. The concern is about external validity, not internal consistency. A fresh-attack evaluation would settle it, so the conditional verdict remains appropriate. I would not reject the paper, but the conclusion should be tempered until the transfer to unseen human attacks is demonstrated.","tokens_in":13529,"tokens_out":3179,"duration_ms":36975,"concrete_test":"Run a fourth, fully held-out break-it round: deploy A3 (or A4), ask new crowdworkers to fool it under the same instructions, collect roughly 1000 fresh offensive examples, and evaluate A3 and S3 on this entire batch without training on any of those examples. If A3's advantage over S3 on this fresh batch approaches the roughly 26-point all-round gap in Table 6, the robustness claim is supported; if the gap shrinks sharply, the reported gains are largely artifacts of training on the same attack distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the build-it, break-it, fix-it loop yields models more robust to human adversarial attack—rests on Table 6. But each adversarial round's test set is a random split of the very examples crowdworkers produced to fool A0 and A_{i-1} (§4.1, §5.2.2). The round-i comparison is therefore partly definitional: A_{i-1} scores 0 on round i by design, and A_i has trained on the other 90% of the same attack distribution. Standard supervised generalization to a held-out split of the same attack batch is not the same as robustness to attacks generated against the deployed model after training. The A_i-vs-S_i gap conflates 'trained on examples of this attack type' with 'inherently harder to fool.' The abstract's claim of being 'considerably more robust than previous systems' is also broader than the evidence: no comparison to Mechanical Turker Descent (the closest prior method cited in §2) or to any independent held-out adversary is reported. This does not invalidate the collected data or the multi-turn context finding, but it narrows the robustness claim's scope substantially.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an iterative 'build it, break it, fix it' training scheme for offensive language detection in dialogue. Crowdworkers are asked to submit offensive utterances that a current classifier labels safe; the collected examples are added to the training data, and the process is repeated over several rounds. The authors collect three rounds of single-turn adversarial examples and one round of multi-turn adversarial examples, together with matched standard (non-adversarial) collections. Using BERT-based and fastText classifiers, they report that adversarially trained models outperform standard-trained models on the adversarial test sets, that later adversarial rounds contain less profanity and more negation and figurative language, and that for BERT-based models adding dialogue context improves multi-turn detection, with the best variant reaching 66.4 OFFENSIVE-class F1. Code, data, and trained models are promised for release.","tokens_in":13815,"tokens_out":3419,"duration_ms":38012,"significance":"If the central robustness claim holds, this is a useful empirical contribution to dialogue safety: it provides a practical human-in-the-loop data collection procedure, a new adversarial evaluation suite, and evidence that dialogue context matters for offensive language detection. The multi-turn experiments are run with five seeds and show consistent context gains for BERT-based models, and the distributional analysis of adversarial versus standard examples is informative. However, the paper's strongest claim—that the resulting models are 'considerably more robust than previous systems'—is broader than the evidence supports, because the evaluation protocol measures generalization to a random split of attacks collected against the deployed predecessor, not to a fresh, held-out attack round from an independent adversary.","major_comments":[{"comment":"The central robustness comparison is measured on same-round random splits of adversarial examples, and the paper's own statement that 'all scores of 0 in Table 6 are by design' confirms the issue: for round i, the test examples were collected specifically to fool A0 and A_{i-1}, so A_{i-1} scores 0 by construction and A_i has trained on the other 90% of that same attack batch. This conflates supervised generalization to a held-out split of one attack distribution with robustness to unseen human attacks in deployment. To support the abstract's claim of being 'considerably more robust than previous systems,' the authors should either collect an additional attack round against the final deployed model (or use an independent adversary) and report performance on that held-out round, or substantially temper the robustness claim to 'robust to the specific attack distributions encountered during iterative collection.'","section":"§5.2.2, Table 6"},{"comment":"The abstract claims the approach is 'considerably more robust than previous systems,' but no comparison is made to any prior method beyond the WTC-trained baseline A0 and the standard-trained models Si. In particular, Mechanical Turker Descent (Yang et al., 2018) is cited in §2 as closely related, yet no comparison to that approach or any other adversarial-collection baseline is reported. The claim should be made relative to the actual baselines evaluated, or a comparison to a prior method should be added.","section":"Abstract and §2"},{"comment":"Single-turn results are reported without error bars or multiple seeds. Since BERT fine-tuning is stochastic and the adversarial test sets contain only 300 OFFENSIVE examples per round, the numerical gaps in Table 6 (e.g., round 2: S3 31.8 vs. A3 62.1) could be accompanied by variance estimates or significance tests. Without these, it is difficult to assess whether the round-by-round improvements are statistically reliable, even though the qualitative pattern is consistent.","section":"§5.2.2, Table 6"}],"minor_comments":[{"comment":"There are several typos and spacing errors, including 'Crowderworker' in §4.1, 'langauge' in §1, 'thestandard task' in the caption of Table 5, and 's single-turn' in §6.1.","section":"Throughout"},{"comment":"The main text says the points 'have no other meaning (e.g. no monetary value),' but Appendix B states that users are 'provided with bonuses for good effort.' Please clarify whether and how effort was rewarded, since this affects the characterization of the gamification.","section":"§4.1 and Appendix B"},{"comment":"The multitasking mixing parameter and the final classification bias are both tuned on the validation sets, but the paper does not report the chosen values or sensitivity to them. A sentence stating the selected values or noting that results were robust to their variation would improve reproducibility.","section":"§5.1.2"},{"comment":"Table 7 repeats the average score information also given in Table 11; consider merging or cross-referencing to avoid redundancy.","section":"Table 7 and Table 11"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution to dialogue safety, and the multi-turn context finding is well supported. My main reservation is scope: the headline robustness claim is stronger than the evaluation design supports. I would encourage the editors to request either an additional held-out attack round or a revised framing that explicitly limits the claim to the iterative collection protocol. The absence of any comparison to prior adversarial-collection methods (especially Mechanical Turker Descent) should also be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth a serious look, but the headline robustness claim is weaker than the abstract suggests. The build-break-fix loop for dialogue safety is a genuine contribution, and the multi-turn context result is solid. The main soft spot is the evaluation of adversarial robustness, which relies on same-round test splits.\n\nWhat's new: applying the build-break-fix idea to offensive language detection, with crowdworkers as breakers and an iterative retraining loop. The paper collects and will release adversarial and standard datasets, and shows that adversarial data differs in interesting ways (less profanity, more negation and figurative language). The multi-turn experiments are well done: five runs, standard deviations, and clear evidence that supplying conversation context improves BERT-based classifiers, with the segment-embedding variant reaching 66.4 offensive-class F1. These findings are useful to anyone working on content moderation or safe dialogue.\n\nWhere it gets soft: Table 6 is the center of the robustness claim, but the test sets are random splits of the very examples crowdworkers produced to fool the deployed model. Round i's adversarial test set is the held-out part of attacks aimed at A0 and A_{i-1}. So A_i has effectively trained on the same attack distribution as the test set, and the zero scores for A0 and A_{i-1} are definitional. The A_i-vs-S_i gap therefore conflates 'exposed to this attack type' with 'intrinsically harder to fool.' To support the abstract's 'considerably more robust than previous systems,' you'd want a fresh attack round from a new set of breakers, or at least a comparison against Mechanical Turker Descent, the closest prior method. The single-turn results also lack error bars, which is annoying given the multi-turn care.\n\nNone of this invalidates the collected data or the context finding. It does narrow the robustness claim. This is a conditional accept situation, not a reject.\n\nWho is this for: NLP researchers working on adversarial robustness, content moderation, and safe dialogue systems. The datasets and method description are valuable even if the evaluation is incomplete. I'd cite the multi-turn finding and use the dataset in future work. Send it to review; a good referee should push for a cleaner holdout attack round.","headline":"A useful adversarial dataset and a solid multi-turn context result, but the single-turn robustness claim rests on same-round test splits and is overstated as 'considerably more robust.'","tokens_in":14262,"tokens_out":2214,"would_cite":true,"duration_ms":20570,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An iterative crowdworker attack-and-retrain loop yields substantially more robust offensive-language classifiers, and dialogue context is necessary to catch offensive replies.","keywords":["dialogue safety","offensive language detection","adversarial human attack","build it break it fix it","crowdsourcing","BERT","dialogue context","robustness"],"falsifier":"Collect a fresh set of 1,000 offensive messages from crowdworkers who are never shown the classifier (or from a different crowd pool), withhold them, retrain $A_i$ and $S_i$ on their own rounds, and compare OFFENSIVE-class $F_1$ on that independent set. If $A_i$ does not beat $S_i$ there, the central robustness claim fails; the paper's Table 6 numbers are computed on test splits drawn from the same round's break-it collection, so the independent set is a direct check.","tokens_in":13270,"feed_emoji":"🛡️","tokens_out":13722,"duration_ms":113964,"temperature":0.7,"pith_summary":"An iterative build-it-break-it-fix-it loop, with human crowdworkers as the breakers, can make dialogue-safety classifiers meaningfully more robust to adversarial offensive language. The paper starts from a BERT-based classifier (a large pre-trained transformer language model) trained on a public toxic-comments dataset, then asks crowdworkers to submit offensive messages the current model marks as safe; each round, the model is retrained on all prior broken examples. Across three single-turn rounds, the adversarially trained models $A_1$ through $A_3$ beat standard-data models $S_1$ through $S_3$ on every adversarial test set, with $A_3$ reaching 67.6 OFFENSIVE-class $F_1$ over all rounds. The paper further establishes that offense in dialogue is context-dependent: a reply can be innocuous alone but hurtful after certain history, and the best model, which feeds the dialogue history and the response as separate segments into BERT, reaches 66.4 OFFENSIVE-class $F_1$ on multi-turn attacks, well above models that ignore context. This matters because deployed conversational agents and forums are attacked by humans who actively seek the classifier's blind spots, while most benchmark datasets evaluate single sentences in isolation.","feed_headline":"Retraining on attacks beats standard models for offensive dialogue","feed_subtitle":"Each round, crowdworkers try to fool the model; retraining on their misses catches 67.6 percent of offensive replies.","key_machinery":"Two mechanisms carry the argument. The first is the build-it-break-it-fix-it loop itself: human crowdworkers are asked to \"beat the system\" by producing messages the current model calls SAFE but the worker considers offensive, and every successfully breaking example is added to the next round's training data. The loop is parameter-free: no gradient-based adversary is needed, only human judgment. The second is the dialogue-segment input representation: a BERT-based classifier in which the conversation history and the final utterance are stored as two separate segments, each with its own segment embedding, so the transformer can condition the SAFE/OFFENSIVE decision on what was said before. The evaluation metric is OFFENSIVE-class $F_1$, the harmonic mean of precision and recall on the offensive class, because the dataset skews heavily SAFE.","core_discovery":"The paper's central claim is that adversarial robustness in offensive-language detection can be learned by repeatedly exposing a classifier to human adversaries and retraining on what defeats it. Starting from $A_0$, a BERT-based binary classifier trained on a public toxic-comments corpus, the loop collects 1,000 crowdworker submissions per round that both $A_0$ and the previous round's model $A_{i-1}$ label SAFE but the worker judges OFFENSIVE. A new model $A_i$ is trained on all previous adversarial rounds plus the original corpus. The paper reports that on each adversarial round's test split the adversarial models outperform the standard models trained on non-adversarial offensive examples: $A_3$ scores 67.6 OFFENSIVE-class $F_1$ across rounds 1-3, versus 41.8 for $S_3$. The second claim is that dialogue context is not decorative but load-bearing: a BERT variant that encodes the conversation history and final response as separate segments reaches 66.4 OFFENSIVE-class $F_1$ with context versus 59.0 without, and the fastText baseline gets worse when context is added (23.6 with context, 37.1 without). The paper interprets this as evidence that context helps only when the architecture can represent it.","pith_inferences":["Beyond the paper: because each round's test split is drawn from the same crowdworker attack distribution that produced the training examples, the reported $F_1$ numbers likely overstate robustness against an independent attacker who never saw the model; a cleaner test would collect a fresh round of attacks after training stops and evaluate all models on that held-out set.","Beyond the paper: the same loop could be applied to other dialogue-safety targets, such as detecting manipulation, harassment, or unsafe generative responses, and might be combined with algorithmic adversarial attacks to cover attack styles humans do not find.","Beyond the paper: if context is as load-bearing as the multi-turn results suggest, then single-sentence safety benchmarks underestimate real-world difficulty, and safety evaluation should adopt contextual test suites like this one.","Beyond the paper: the 0.0 scores of $A_0$ and $A_{i-1}$ on their own collected rounds do not mean these models forgotten offense; they mean the test examples were selected precisely because those models classified them as SAFE, so the comparison across models in the same round is a comparison on attack styles aimed at the previous models, not on a fixed difficulty scale."],"forward_implications":["Models trained on adversarial attacks generalize to previously unseen attacks collected in the same round: $A_3$ reaches 67.6 OFFENSIVE-class $F_1$ over all single-turn adversarial rounds, while the standard-data model $S_3$ reaches only 41.8.","The benefit compounds with iteration: scores on round 3 adversarial test move from 32.1 for $A_1$ to 59.9 for $A_3$, while the standard models stay near 14.","Dialogue context matters for offense detection: BERT with dialogue segments and context scores 66.4 OFFENSIVE-class $F_1$ on the multi-turn task, versus 59.0 for the same architecture without context; a fastText bag-of-embeddings model actually degrades when context is added.","Adversarial collection changes the data distribution: the share of profanity-laden examples drops from 18.2% in standard collection to 1.2% by adversarial round 3, while reliance on negation, figurative language, and world knowledge increases.","The iterative retraining maintains reasonable performance on the original benchmark: all adversarial models stay within a few points of the baseline on the toxic-comments test set, with only a small trade-off observed in round 3."],"supporting_citations":[{"why":"Created the toxic-comments dataset from talk-page edit discussions, the bootstrapping training corpus for $A_0$.","marker":"(Wulczyn et al., 2017)"},{"why":"Released the seven-label toxic-comment benchmark and competition that defines the data split and task used to train the baseline.","marker":"(Google, 2018)"},{"why":"Provides the BERT transformer architecture and pretrained weights underlying all strong classifiers in the paper.","marker":"(Devlin et al., 2018)"},{"why":"Provides the fastText bag-of-embeddings classifier used as the weak non-contextual baseline.","marker":"(Joulin et al., 2017)"},{"why":"Provides the OFFENSIVE/SAFE grouping and the BiLSTM baseline whose weighted-$F_1$ score the paper compares against on the toxic-comments benchmark.","marker":"(Khatri et al., 2018)"},{"why":"Originated the build-it-break-it-fix-it contest methodology for secure software, which this paper adapts to classifier training.","marker":"(Ruef et al., 2016)"},{"why":"Brings the build-it-break-it idea to NLP shared tasks, the direct precursor of this human-in-the-loop scheme.","marker":"(Ettinger et al., 2017)"},{"why":"Provides the game-with-a-purpose crowdworker method that motivates the scoring incentive used during break-it collection.","marker":"(Yang et al., 2018)"},{"why":"Supplies the chit-chat dialogues used to source SAFE examples and conversation contexts for the multi-turn task.","marker":"(Dinan et al., 2019)"},{"why":"Shows that classifiers can be deceived by negation, which motivates the paper's analysis of negation in adversarial examples.","marker":"(Hosseini et al., 2017)"}],"fun_headline_variants":["Human-in-the-loop retraining beats baselines on offensive dialogue","Adversarial human attack loop boosts offensive F1 from 41.8 to 67.6","Context-aware training lifts offensive dialogue F1 to 66.4","Build-break-fix with human attacks beats standard offensive models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central robustness claim assumes that crowdworker attacks collected against the exact deployed model in the same round are a faithful proxy for the attacks the system will encounter in the real world, so beating those same-round test examples means the system is genuinely safer.","fun_headline_variants_meta":{"raw":{"variants":["Human-in-the-loop retraining beats baselines on offensive dialogue","Adversarial human attack loop boosts offensive F1 from 41.8 to 67.6","Context-aware training lifts offensive dialogue F1 to 66.4","Build-break-fix with human attacks beats standard offensive models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001373,"raw_usage":{"total_tokens":5581,"prompt_tokens":979,"completion_tokens":4602,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":4531}},"tokens_in":595,"tokens_out":4602,"duration_ms":27958,"temperature":1.0,"reasoning_tokens":4531,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:49:16.737603+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a fresh set of 1,000 offensive messages from crowdworkers who are never shown the classifier (or from a different crowd pool), withhold them, retrain $A_i$ and $S_i$ on their own rounds, and compare OFFENSIVE-class $F_1$ on that independent set. If $A_i$ does not beat $S_i$ there, the central robustness claim fails; the paper's Table 6 numbers are computed on test splits drawn from the same round's break-it collection, so the independent set is a direct check.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Released the seven-label toxic-comment benchmark and competition that defines the data split and task used to train the baseline."},{"cited_title":"Detecting Offensive Content in Open-domain Conversations using Two Stage Semi-supervision","cited_arxiv_id":"1811.12900","evidence_quote":"Provides the OFFENSIVE/SAFE grouping and the BiLSTM baseline whose weighted-$F_1$ score the paper compares against on the toxic-comments benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Originated the build-it-break-it-fix-it contest methodology for secure software, which this paper adapts to classifier training."},{"cited_title":"Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Task","cited_arxiv_id":"1711.01505","evidence_quote":"Brings the build-it-break-it idea to NLP shared tasks, the direct precursor of this human-in-the-loop scheme."},{"cited_title":"Miller, Arthur Szlam, Douwe Kiela, and Jason Weston","cited_arxiv_id":null,"evidence_quote":"Provides the game-with-a-purpose crowdworker method that motivates the scoring incentive used during break-it collection."}],"review_version":1}