Robustness to Modification with Shared Words in Paraphrase Identification

Minlie Huang; Zhouxing Shi

REVIEW 3 major objections 3 minor 35 references

Robustness to Modification with Shared Words in Paraphrase Identification

T0 review · 3 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Paraphrase-identification models are fragile when words shared by both sentences change, and adversarial training repairs much of the damage.

desk verdict A genuinely new attack for paraphrase models with a real fragility signal, but the drop magnitudes are not fully nailed down because label validity is only spot-checked. read the letter →

arxiv 1909.02560 v5 pith:UBGNBEVO submitted 2019-09-05 cs.CL

Zhouxing Shi , Minlie Huang This is my paper · ORCID

classification cs.CL

keywords paraphraseidentificationadversarialexamplessharedwordsrobustnessBERTtrainingbeamsearchquestionpairs

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Paraphrase identification is the task of deciding whether two sentences mean the same thing. The paper claims that modern models, including BERT, are fragile when words shared by both sentences are changed, even when the paraphrase relation is preserved. On modified examples built from Quora Question Pairs, BERT's accuracy falls from 94.3% to 24.1%, and on MRPC from 97.0% to 23.8%. The same construction also makes negative pairs look positive by injecting shared content words into unrelated templates. Training on a small share of such modified examples restores much of the lost accuracy, suggesting the failure is correctable rather than intrinsic.

What carries the argument

The load-bearing object is the shared-word modification: a pair of positions, one in each sentence, is replaced by a word that appears in both sentences. For positive examples, a shared word in both sentences is replaced by one new shared word, preserving the paraphrase; for negative examples, new shared words are introduced into two unrelated sentences, which still leaves them non-paraphrases. A constrained beam search chooses which positions and which substitution words to use, and BERT's masked language model both supplies candidates and scores their compatibility with both masked contexts. The mechanism isolates the model's sensitivity to lexical overlap from its ability to judge semantic equivalence.

What would settle it

Collect a large sample of modified examples (thousands, not hundreds) on which the target model flips, and have expert annotators judge the original label under a strict protocol; if the disagreement rate climbs well above the reported 12-21%, the measured accuracy drop partly reflects label-changing edits rather than model fragility, while disagreement near that level confirms the vulnerability.

Watch

Extended reading notes

Core claim

The central claim is that paraphrase-identification models rely on shared vocabulary more than on meaning. For positive pairs, replacing a word that appears in both sentences with the same new word keeps the pair a paraphrase (e.g., switching "purpose of life" to "measure of value") but makes the target model predict negative. For negative pairs, replacing a noun, verb, or adjective in each sentence so that the two sentences acquire new shared content words flips the prediction to positive. The authors generate such examples with a two-stage beam search scored by the target model's loss, using BERT's masked language model to propose substitutions that fit both contexts. Human annotation of a 100-example sample from each dataset indicates the generated pairs mostly keep their original labels (79-88% agreement), so the accuracy collapse is not simply the result of changing the label.

Load-bearing premise

The whole accuracy-drop measurement rests on the assumption that the generated sentence pairs really do keep their original paraphrase labels; the paper only spot-checks this on 100 examples per dataset, where human raters agree with the original label only 79% to 88% of the time.

Editorial extensions

If this is right

On adversarially modified examples, all three tested architectures—BiMPM, DIIN, and BERT—drop to near or below chance on at least one dataset (e.g., BiMPM on MRPC falls to 1.7% overall).
The failure is bidirectional: positive examples become false negatives and negative examples become false positives, so both paraphrase and non-paraphrase decisions are affected.
Adversarial training with about 10% modified examples per batch improves accuracy on modified examples substantially (BERT on QQP from 24.1% to 66.0%; on MRPC from 23.8% to 87.0%) while leaving accuracy on the original test sets nearly unchanged.
Because the perturbations are not semantic-preserving, they reveal a distinct vulnerability from PAWS word-order adversaries, which only tested negative examples with identical bags of words.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same generation recipe could be applied to other sentence-pair tasks (duplicate detection, retrieval, natural language inference) to measure how much of their performance is lexical-overlap bias; the paper does not test those tasks.
The recovery pattern in the tables suggests adversarial training is more effective on negative examples (BERT QQP negative accuracy rises from 14.8% to 79.0%) than on positive examples (33.4% to 53.0%), so future defenses may need to spend extra effort on positive shared-word replacements.
If the apparent fragility is real, then paraphrase benchmarks with naturally varying word overlap may overstate model quality; an explicit diagnostic controlling overlap would make this bias measurable.
The small human-validation sample leaves room for a stricter test; an annotator disagreement rate much larger than the reported 12-21% would require revising the size of the claimed accuracy drop.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know about this paper: it studies a genuinely new failure mode for paraphrase identification. Instead of semantic-preserving edits, the attack replaces or introduces shared content words between the two sentences, which is different from PAWS word scrambling and covers positive examples too. The BERT-masked-LM joint substitution candidate generation is a neat, concise trick, and the reported accuracy drops are dramatic (BERT on QQP from 94.3% to 24.1%). The paper also shows adversarial training recovers a good chunk of the drop, which is a useful practical result.

What is actually good: the method is simple and reproducible, tested on three architecture families (BiMPM, DIIN, BERT), and the authors do report manual evaluation, which is more than many attack papers bother with. The qualitative finding—that shared-word modifications expose real fragility—is credible and worth knowing about.

Now the soft spots. The label-validity evidence is the weakest link. The manual check covers only 100 modified examples per dataset, only for BERT as target, and only from the subset where BERT made mistakes. On that biased sample, human agreement with the original label is 79% on QQP and 88% on MRPC, versus 92%/95% on original examples. So a fifth of the QQP examples used to demonstrate the drop may actually be mislabeled, and the paper cannot estimate the mislabeling rate on the full 1,000-example test set. That doesn't erase the finding, but it means the headline numbers are likely overstated. Also missing: error bars, a random-substitution control, and human validation for the BiMPM/DIIN results. The unresolved citation placeholder '(??)' in Sec. 3.2 is a minor but real blemish.

Who is this for? People working on adversarial robustness, sentence-pair models, or evaluation methodology. It deserves a serious referee, and I would send it back for a round of revisions: report variance, add a random-control baseline, and expand human evaluation to all models and to examples where the model is correct. The central claim that paraphrase models are fragile under shared-word modification holds up in broad strokes; the precise magnitudes are not yet established.

Recommendation: engage with it. Cite it with care, and don't repeat the 24.1% without noting the label-noise caveat.

Referee Report

3 major / 3 minor

Summary. The paper studies robustness of paraphrase identification models under a new type of modification: changing words that are shared by the two sentences (for positive examples) or introducing new shared words (for negative examples). The proposed algorithm samples original examples, restricts replaceable positions by heuristic rules (no stopwords; shared words for positives; same POS for negatives), generates candidate substitutions with a BERT masked language model, and uses beam search to find modifications that increase the target model's loss. The authors evaluate BiMPM, DIIN, and BERT on QQP and MRPC, reporting large accuracy drops on the modified examples (e.g., BERT on QQP from 94.3% to 24.1%) and improved accuracy after adversarial training with modified examples. A manual evaluation on 100 BERT-failure examples per dataset reports human label agreement of 79% (QQP) and 88% (MRPC) on modified examples.

Significance. If label preservation in the modified examples is adequately established, the paper makes a useful contribution. The shared-word modification perspective is genuinely different from PAWS-style word scrambling, and the finding that several architectures, including BERT, can be moved from confident correct predictions to confident wrong predictions by coordinated word replacement is practically important. The adversarial-training result, showing partial recovery without large original-data degradation, is a constructive step. The method is simple, reproducible in principle, and the paper includes several concrete examples. The main weakness is that the validity of the measured accuracy drops rests almost entirely on a small, biased manual evaluation; the quantitative claims, as currently supported, are provisional.

major comments (3)

[Section 4.4, Table 2] The manual evaluation is the only evidence that the modified examples preserve the original labels, but it samples only 100 examples per dataset, exclusively from the subset where BERT makes wrong predictions, and only for BERT as the target model. Human agreement with the original label is 79% on QQP and 88% on MRPC, so 21% and 12% of the sampled modified examples are not confirmed to preserve the label. Because the sample is conditioned on adversarial failure, it cannot estimate the label-preservation rate in the 1,000-example sets used for Table 1, and no validation is reported for BiMPM or DIIN. Since all accuracies in Table 1 and the adversarial-training rows are scored against the original labels, a mislabel rate of this magnitude could materially inflate the reported drops; for example, BERT's QQP accuracy of 24.1% from Table 1 would be substantially higher if a nontrivial fraction of the modified examples are actually incorrectly labeled. The paper should validate labels on a random sample of the full modified test sets (or per model), or report accuracies corrected for label noise.
[Section 4.2, Table 1] The experiments do not include a random-substitution control. Table 1 compares model accuracy on original examples with accuracy on adversarially modified examples, but it never compares against modifications using the same replacement positions and the same BERT-MLM candidate words without optimizing the target-model loss. Without such a control, the large accuracy drops cannot be attributed specifically to the introduction or replacement of shared words; they could in part reflect any lexical substitution or the grammaticality degradation reported in Section 4.4. Adding a control where substitutions are random (or chosen to minimize loss) would directly support the paper's central claim about shared words as the source of fragility.
[Table 1] All results are single point estimates on 1,000 sampled examples, with no standard errors, confidence intervals, or multiple random seeds. Given the 12-21% label disagreement in the manual evaluation and the fact that some reported differences are small (e.g., QQP adversarial-training gains for BiMPM: 11.1% to 21.4%), the statistical stability of the comparisons is unclear. The paper should report bootstrap intervals over the sampled examples or variance across training/generation seeds for the main accuracy columns.

minor comments (3)

[Section 3.2] The phrase "some recent works ( ??)" is an unresolved placeholder and should be replaced with the actual citations or removed before publication.
[Appendix A] The hyperparameters S, K, and B are set to 5, 25, and 25 on QQP and doubled on MRPC, but no sensitivity analysis or rationale is given; a sentence explaining why these values were chosen and whether results are stable to them would strengthen the paper.
[Section 4.4] The manual evaluation reports only accuracy and grammaticality averages; reporting annotator-level agreement (e.g., Fleiss' kappa) would help quantify how difficult the label judgment is, especially given the low agreement on QQP modified examples.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the attack-by-construction nature is transparent and the central robustness claim is anchored by external human evaluation.

full rationale

The paper's modified examples are generated by an explicit adversarial search (Sec. 3.3: beam search minimizes the gold-label score until the target model predicts incorrectly), so the low accuracies in Table 1 are partly a consequence of the search objective. This is not, however, a circular derivation in the sense prohibited here: the central claim is existential — that valid paraphrase/non-paraphrase pairs with shared-word modifications exist and cause model failure — and that existence is independently checked by human annotation in Sec. 4.4, which reports 79% and 88% label agreement on 100-example BERT-failure samples. The finite human disagreement and the biased sampling of only model-failure examples are validity and magnitude concerns, not circularity. No parameter is fitted to a subset and then renamed as a prediction, no load-bearing self-citation or imported uniqueness theorem is used, and no ansatz is smuggled in via citation. The method is evaluated on standard external benchmarks (QQP, MRPC) with open-source target models and an external pre-trained BERT language model. The incomplete citation '(??)' in Sec. 3.2 is a completeness issue, not a circularity issue. Overall, the derivation chain is self-contained and the robustness claim has independent empirical content.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities. The central claim rests on a few hand-set hyperparameters and on the assumption that the heuristic modification rules preserve labels, which is only weakly validated.

free parameters (3)

Step number limit S = 5 for QQP, 10 for MRPC
Hand-set limit on how many word pairs can be modified in one search; doubling for MRPC is ad hoc and could affect the success rate of the attack.
Candidate substitution set size K = 25 for QQP, 50 for MRPC
Number of top masked-language-model words considered per position; hand-chosen and not swept, which may influence modification quality.
Beam size B = 25 for QQP, 50 for MRPC
Beam width in the search; larger values would find stronger attacks but cost more compute. The choice affects the reported accuracy drops.

assumptions (3)

domain assumption Heuristic constraints on replaceable positions preserve the gold label and language quality.
Section 3.2 states that stopwords are not replaced, positive examples require shared words, and negative examples require same POS. The paper assumes these rules keep the modified example in the same class, but manual evaluation shows this only holds for 79% to 88% of cases.
domain assumption The product of two BERT masked language model probabilities approximates a word's compatibility with both sentence contexts.
In Section 3.2, the joint probability is factorized as P(w|context1)*P(w|context2). This assumes independence of the two contexts, which may not hold, but is a reasonable approximation for generation.
domain assumption Adversarial examples need not be semantic-preserving to be valid evidence of a robustness issue.
The paper explicitly departs from semantic-preserving perturbations in Section 1. This is a design choice, but it means the validity of the attack depends entirely on human judgment of label preservation, which is only partially verified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Robustness to Modification with Shared Words in Paraphrase Identification." pith.science (2026). https://pith.science/paper/UBGNBEVO

@misc{pith2026190902560,
  author       = {Pith},
  title        = {Pith review of: Robustness to Modification with Shared Words in Paraphrase Identification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UBGNBEVO}},
  note         = {Machine review of arXiv:1909.02560}
}

read the original abstract

Revealing the robustness issues of natural language processing models and improving their robustness is important to their performance under difficult situations. In this paper, we study the robustness of paraphrase identification models from a new perspective -- via modification with shared words, and we show that the models have significant robustness issues when facing such modifications. To modify an example consisting of a sentence pair, we either replace some words shared by both sentences or introduce new shared words. We aim to construct a valid new example such that a target model makes a wrong prediction. To find a modification solution, we use beam search constrained by heuristic rules, and we leverage a BERT masked language model for generating substitution words compatible with the context. Experiments show that the performance of the target models has a dramatic drop on the modified examples, thereby revealing the robustness issue. We also show that adversarial training can mitigate this issue.

Figures

Figures reproduced from arXiv: 1909.02560 by the authors.

**Figure 1.** Examples with labels positive and negative respectively, originally from Quora Question Pairs (QQP) (Iyer et al., 2017). “(P)” and “(Q)” are original sentences while “(P’)” and “(Q’)” are modified. Modified words are highlighted in bold. “Output” indicates the change of output labels by BERT (Devlin et al., 2018), where the percentage numbers are confidence scores. tences change from asking about “purpose of life”… view at source ↗

**Figure 2.** Examples of identifying replaceable position [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 20 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890--2896

work page 2018
[4]

Petr B e lohl \'a vek, Ond r ej Pl \'a tek, Zden e k Z abokrtsk \`y , and Milan Straka. 2018. Using adversarial examples in natural language processing. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)

work page 2018
[5]

O'Reilly Media, Inc

Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. " O'Reilly Media, Inc."

2009
[6]

Bowman, Gabor Angeli, Christopher Potts, and Christopher D

Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics

2015
[7]

Paul Clough. 2000. Plagiarism in natural and programming languages: an overview of current tools and technologies

work page 2000
[8]

Alexis Conneau, Douwe Kiela, Holger Schwenk, Lo \" c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670--680

work page 2017

Show all 35 references

[9]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

2018 arXiv
[10]

William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005)

2005
[11]

Chaoqun Duan, Lei Cui, Xinchi Chen, Furu Wei, Conghui Zhu, and Tiejun Zhao. 2018. Attention-fused deep matching network for natural language inference. In IJCAI, pages 4033--4040

2018
[12]

Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. Hotflip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31--36

2018
[13]

Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650--655

2018
[14]

Yichen Gong, Heng Luo, and Jian Zhang. 2017. Natural language inference over interaction space. arXiv preprint arXiv:1709.04348

2017 arXiv
[15]

Yu-Lun Hsieh, Minhao Cheng, Da-Cheng Juan, Wei Wei, Wen-Lian Hsu, and Cho-Jui Hsieh. 2019. On the robustness of self-attentive models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1520--1529

2019
[16]

Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In Advances in neural information processing systems, pages 2042--2050

2014
[17]

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700--4708

2017
[18]

Shankar Iyer, Nikhil Dandekar, and Kornél Csernai. 2017. First quora dataset release: Question pairs

2017
[19]

Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human La...

2018
[20]

Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021--2031

2017
[21]

Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2019. Is bert really robust? natural language attack on text classification and entailment. arXiv preprint arXiv:1907.11932

2019 arXiv
[22]

Seonhoon Kim, Jin-Hyuk Hong, Inho Kang, and Nojun Kwak. 2018. Semantic sentence matching with densely-connected recurrent and co-attentive information. arXiv preprint arXiv:1805.11360

2018 arXiv
[23]

Volodymyr Kuleshov, Shantanu Thakoor, Tingfung Lau, and Stefano Ermon. 2018. Adversarial examples for natural language classification problems

2018
[24]

Pasquale Minervini and Sebastian Riedel. 2018. Adversarially regularising neural nli models to integrate logical background knowledge. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 65--74

2018
[25]

Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text matching as image recognition. In Thirtieth AAAI Conference on Artificial Intelligence

2016
[26]

Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1085--1097

2019
[27]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856--865

2018
[28]

Suranjana Samanta and Sameep Mehta. 2017. Towards crafting text adversarial samples. arXiv preprint arXiv:1707.02812

2017 arXiv
[29]

Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Y Ng. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in neural information processing systems, pages 801--809

2011
[30]

Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17 , pages 4144--4150

2017
[31]

Puyudi Yang, Jianbo Chen, Cho-Jui Hsieh, Jane-Ling Wang, and Michael I Jordan. 2018. Greedy attack and gumbel attack: Generating adversarial examples for discrete data. arXiv preprint arXiv:1805.12316

2018 arXiv
[32]

Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013--2018

2015
[33]

Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. Paws-x: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat...

2019
[34]

Huangzhao Zhang, Hao Zhou, Ning Miao, and Lei Li. 2019 a . Generating fluent adversarial examples for natural languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5564--5569

2019
[35]

Yuan Zhang, Jason Baldridge, and Luheng He. 2019 b . Paws: Paraphrase adversaries from word scrambling. arXiv preprint arXiv:1904.01130

2019 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890--2896

work page 2018

[4] [4]

Petr B e lohl \'a vek, Ond r ej Pl \'a tek, Zden e k Z abokrtsk \`y , and Milan Straka. 2018. Using adversarial examples in natural language processing. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)

work page 2018

[5] [5]

O'Reilly Media, Inc

Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. " O'Reilly Media, Inc."

2009

[6] [6]

Bowman, Gabor Angeli, Christopher Potts, and Christopher D

Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics

2015

[7] [7]

Paul Clough. 2000. Plagiarism in natural and programming languages: an overview of current tools and technologies

work page 2000

[8] [8]

Alexis Conneau, Douwe Kiela, Holger Schwenk, Lo \" c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670--680

work page 2017