Deceiving Question-Answering Models: A Hybrid Word-Level Adversarial Approach

Jiyao Li; Mingze Ni; Wei Liu; Yongshun Gong

REVIEW 5 major objections 4 minor 73 references

Deceiving Question-Answering Models: A Hybrid Word-Level Adversarial Approach

T0 review · 5 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read QA-Attack claims that fusing attention weights with deletion-based importance scores finds the context words whose synonym replacement flips a question-answering model's answer, for both informative and boolean questions.

desk verdict A plausible and simple QA attack with a new attention-deletion fusion, but the comparative claim is undercut by an unfair TextFooler baseline and missing error bars. read the letter →

arxiv 2411.08248 v1 pith:K4QSDT7V submitted 2024-11-12 cs.CL cs.AIcs.LG

Jiyao Li , Mingze Ni , Yongshun Gong , Wei Liu This is my paper · ORCID

classification cs.CLcs.AIcs.LG

keywords adversarialattackquestionansweringword-levelperturbationattentionmechanismdeletionrankinghybridfusionBERTmaskedlanguagemodelrobustness

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Question-answering models remain vulnerable in ways that existing attacks only partially cover: most attack methods handle either informative questions that ask who, what, or where, or yes/no boolean questions, but not both, and they often need many edits or sacrifice fluency. This paper proposes QA-Attack, a word-level attack that treats both question types with one recipe: find the context words a QA model relies on by averaging its attention weights and by measuring how much its answer probability drops when each word is deleted, fuse the two rankings, and replace the top few words with synonyms suggested by a masked language model. The paper claims this hybrid ranking beats five existing attack methods across SQuAD, SQuAD 2.0, NarrativeQA, NewsQA, and BoolQ, on attack success, semantic preservation, fluency, and speed. If right, it shows that a small number of well-chosen word substitutions can reliably flip QA predictions while leaving the passage looking natural.

What carries the argument

The machinery is the pipeline in Algorithm 1. Attention-based Ranking (ABR) averages the attention scores of all layers and heads of the victim's encoder-decoder on the question-context pair, keeps only context-token scores, and normalizes them. Removal-based Ranking (RBR) computes, for each context word $c_i$, the importance $I_i = L_F(a \mid q, C) - L_F(a \mid q, C \setminus c_i)$, meaning the drop in answer log-probability when that word is deleted. The Hybrid Ranking Fusion adds the two normalized scores per word, selects the top-$k$ words, generates $d$ synonym candidates for each by masking the word and asking BERT's masked language model, and chooses the candidate with the largest logit gap that flips the answer. The load-bearing idea is that attention and deletion measure different failure modes, so their sum finds words neither method alone would put first.

What would settle it

Run QA-Attack, then repeat with the attention score for every word replaced by a random permutation, or set to zero, while keeping the removal-based ranking unchanged; if attack success does not fall, the attention component of the fusion is not doing the work the paper claims. Alternatively, run the attack on a deployed model that exposes only answers and confidence scores, where the attention-based component cannot be applied at all.

Watch

Extended reading notes

Core claim

The central claim is that Hybrid Ranking Fusion (HRF), which adds each word's normalized attention score from the victim model to its normalized deletion-based importance score, identifies the tokens whose replacement most reliably changes the answer, for informative and boolean questions alike. In the paper's experiments QA-Attack lowers F1 on the T5 SQuAD 1.1 model to 4.67 from an 88.9 baseline, and also produces adversarial contexts with higher ROUGE and BLEU scores and lower semantic similarity to the original context than the baselines it is compared with. The method selects the final adversary from a small candidate list by maximizing the gap between the logits of the original and the perturbed answer, limiting itself to a handful of substitutions, with topk equal to 5 by default.

Load-bearing premise

The method assumes the attacker can access both the victim model's attention weights and its output logits, and that averaged attention actually marks the words whose replacement will fool the model; if a deployed system hides attention, or if attention is uncorrelated with vulnerability, the claimed fusion collapses to deletion-only ranking.

Editorial extensions

If this is right

A single attack configuration works on both informative QA and boolean QA, so defenses cannot assume an attacker will specialize by question type.
With only about five word substitutions per passage, adversarial examples can preserve grammaticality and fluency while changing the model's answer, making the attack hard for a human reader to spot.
Adversarial samples generated against one QA model also degrade other QA models such as RoBERTa, DistilBERT, and MultiQA, so the vulnerabilities are not unique to the attacked architecture.
Light defenses like FGWS and RanMASK reduce but do not eliminate the attack's success, and retraining with adversarial examples helps until those examples exceed roughly 30 percent of the training data.
Larger BERT models are harder to attack than smaller ones, suggesting that model capacity itself contributes to robustness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the fusion step only needs attention weights and output logits, the same recipe could be adapted to other attention-based reading tasks such as fact-checking or natural language inference, where passage words rather than question words are the attack surface; that extension is not tested in the paper.
The paper does not compare HRF against a deletion-only ranking with attention scores replaced by random noise; such an ablation would isolate whether attention contributes real signal or merely correlates with deletion importance. This is an editorial suggestion, not a paper claim.
If attention-based ranking is tied to model internals, then the practical reach of QA-Attack is limited to white-box or open-weight systems; closed APIs that hide attention would only be attackable through the deletion component.
The logit-gap selection criterion implies the attack is strongest when the model's pre-attack confidence is high, so connecting attack success to model calibration could let defenders predict which examples are most at risk.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a plausible word-level attack for QA with a genuinely new, if modest, algorithmic twist, and the attack itself probably works. But the headline claim that it beats all existing attack methods is not backed by the current experiments, mainly because the TextFooler baseline was handicapped.

What's new: HRF, a fusion of attention-based ranking and removal-based ranking for choosing which context words to replace. The components are established—deletion importance, BERT-MLM synonyms, attention-based selection—but the specific combination for QA is new and the algorithm is simple and transparent. The ablation showing HRF beating either ranking alone is the most convincing part. They also ship code.

What it does well: the experimental coverage is broad: five datasets, three victim models, multiple metrics, plus transferability and defense experiments. If the numbers are accurate, the attack is effective: post-attack F1 on T5/SQuAD 1.1 drops to 4.67 from 88.9, consistent with prior adversarial work.

Soft spots, in order of severity. First, the TextFooler adaptation is unfair: the authors removed the question from TextFooler's input. For a QA attack, the question is half the input; TextFooler's word importance is driven by prediction change on the full input, so this handicaps it. The gap on some rows is large (LongT5/SQuAD 1.1: 35.31 vs 7.38), so this is not second-order. The 'surpasses all baselines' claim is not established until a question-aware TextFooler is tested.

Second, no error bars anywhere. Comparative claims are point estimates, and some gaps are small (Table 3 SQuAD 1.1: T3 F1 5.41 vs QA-Attack 4.67), so we cannot judge whether differences are meaningful.

Third, Table 6 has a copy-paste error: the LongT5 rows for TextFooler and T3 are identical to the Bertbase rows.

Fourth, the RobustQA citation [63] points to a POS-tagging robustness paper, not a QA attack.

There's also a scope limitation: ABR needs attention weights, so the method doesn't directly apply to APIs that hide attention. That's a limitation, not a flaw.

Who it's for: adversarial NLP researchers looking for a simple baseline attack for QA robustness benchmarks. It deserves peer review with major revisions: fix the TextFooler baseline, add statistical support, correct the duplicated rows and citation.

Referee Report

5 major / 4 minor

Summary. This paper proposes QA-Attack, a word-level adversarial attack for question-answering systems. Section 3 describes three components: Attention-based Ranking (ABR), which extracts attention scores from the victim transformer's encoder/decoder; Removal-based Ranking (RBR), which scores each context word by the change in output probability when the word is deleted; and Hybrid Ranking Fusion (HRF), which sums normalized scores and attacks the top-k words. Section 3.5 uses a BERT masked language model to generate d synonym candidates per target word, and Section 3.6 selects the candidate that maximizes the logit gap between original and attacked outputs. The experiments in Section 4 cover five datasets (SQuAD 1.1, SQuAD V2.0, NarrativeQA, NewsQA, BoolQ) and three victim model families (T5, LongT5, BERT base), comparing QA-Attack against TASA, TMYC, RobustQA, T3, and an adapted TextFooler, with additional ablation, defense, transferability, and efficiency analyses. The paper claims that QA-Attack deceives QA models and surpasses existing adversarial techniques in success rate, semantics preservation, BLEU score, fluency, and grammar error rate.

Significance. The proposed method is relevant to the adversarial robustness literature for QA. Its strengths include released code, evaluation on both informative and boolean query datasets, and a clean combination of attention- and deletion-based ranking with BERT-MLM synonym substitution. The ablation study in Section 4.5.1 supports the value of the fusion over either ranking alone, and the defense and transfer experiments in Sections 4.8 and 4.9 broaden the empirical scope. However, the headline claim of surpassing all baselines is currently supported only by single point estimates, and one baseline is adapted in a way that removes the question from its input. These issues must be fixed before the comparative claim can be accepted; the core phenomenon that word-level perturbations degrade QA accuracy is consistent with prior work and is not in doubt.

major comments (5)

[Section 4.3] The TextFooler adaptation is unfair and undermines the comparative claim. The text states: 'TextFooler, originally not designed for question-answering tasks, was adapted for our experiments. We modified it to process the context only (questions are removed).' Since TextFooler selects words by the model's prediction change, removing the question removes the query signal from its importance estimates, whereas QA-Attack explicitly uses the question-context pair in ABR and RBR. The gap is large in places, e.g., Table 5, LongT5/SQuAD 1.1 F1: 35.31 for TextFooler vs. 7.38 for QA-Attack, so this is not a second-order effect. Please re-run TextFooler with a question-aware adaptation or remove the 'surpasses baselines' claim.
[Tables 3–6] No variance, confidence intervals, or significance tests are reported for any of the main comparisons. For instance, in Table 3 (T5, SQuAD 1.1), QA-Attack's F1 of 4.67 is compared with T3's 5.41 and RobustQA's 5.89; in Table 5 (LongT5, SQuAD V2.0), QA-Attack's F1 of 27.11 is compared with RobustQA's 29.01. These differences may be real, but with a single evaluation per cell the reader cannot assess whether they are within noise. Please report repeated runs, confidence intervals, or paired significance tests.
[Table 6] The BoolQ results contain duplicated rows that cannot both be correct. The TextFooler row for Bertbase and LongT5 is identical (F1 33.02, EM 11.57, ROUGE 65.11, BLEU 67.81, SIM 8.17), and the T3 row is likewise identical (F1 22.06, EM 11.02, ROUGE 76.17, BLEU 74.62, SIM 6.23). This indicates a copy-paste error and invalidates the BoolQ comparison for those entries; the authors should correct the table and re-run the affected experiments.
[Section 3.2 / Algorithm 1] The attention-ranking procedure is not specified precisely enough to reproduce. Algorithm 1 line 2 writes 'α ← [(c, A(q + C))]' and then filters 'if score ∈ C', while the prose says word-level scores are obtained by summing, averaging, and normalizing attention over all layers and heads. The manuscript should define the exact tensor operations, how question tokens are excluded from the context score, and how ABR is computed for victim models such as BERT base, whose attention is not the T5/UnifiedQA attention described in Section 3.2.
[Section 1 vs. Section 3] The threat model is internally inconsistent. Section 1 criticizes gradient-based methods because 'knowing victim models' internal information is unrealistic in practice,' yet Algorithm 1 takes as input both the victim's logits L and an attention network A for the victim model, and Section 3.6 uses per-token output logits to select candidates. The paper should state the assumed adversary capabilities explicitly and discuss how the method degrades when attention weights are unavailable.

minor comments (4)

[References] Reference [63] is cited for RobustQA, but the bibliography entry is Yasunaga et al., 'Robust multilingual part-of-speech tagging via adversarial training,' which does not appear to be the RobustQA attack described in the text; please verify the citation.
[Tables and figures] There are several typographical issues: 'T able' appears in Tables 1, 3, 4, 5, and 8; Figure 2's caption uses 'HFR' instead of 'HRF'; and Table 8 lists 'NewQA' while the dataset is NewsQA.
[Table 9] Table 9's caption lists 'SIM↑' while Tables 3–5 report SIM as '↓' (lower is better); please clarify the desired direction and keep the notation consistent across tables.
[Section 4.3] The paper alternates between 'Tick Me If You Can' in Section 4.3 and 'Trick Me If You Can' in Section 1; please use one consistent title.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all headline results are measured against external baselines on held-out test sets.

full rationale

QA-Attack's reported numbers (Tables 3-6) are measured attack outcomes on held-out test splits of SQuAD, SQuAD V2, NarrativeQA, NewsQA, and BoolQ, using victim models T5, LongT5, and BERT-base, and are compared with independently published baselines. The algorithm's word-selection (ABR, RBR, HRF) and synonym substitution are search procedures that query the victim model's attention and logits; using the victim model to guide the attack is standard adversarial practice and does not make the measured success rate true by definition. The only self-citations ([30], [62], [65]) appear in related-work, future-work, or generic attention-history contexts and do not supply any empirical result; no uniqueness theorem or ansatz is imported from the authors' prior work. Hyperparameters topk and d are fixed and ablated, not fitted to the test set. The possible unfairness of the TextFooler adaptation (removing the question) is an experimental-validity concern, not a circularity, because QA-Attack's advantage is not constructed by reusing its own output as evidence. No equation in the paper reduces to its inputs by construction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on two types of assumptions: (i) information access assumptions (attention weights and logits available from the victim), and (ii) heuristic assumptions (attention and deletion scores are useful importance proxies; BERT MLM preserves semantics). The hyperparameters topk=5 and d=2 are hand-chosen and the ablation shows stronger attacks at higher values, so the base setting under-represents the method at the cost of fewer modifications.

free parameters (2)

topk = 5
Number of words selected for modification. Section 4.1 sets topk=5; Section 4.5.1 shows higher topk values improve attack success, so the base setting trades off modification rate against effectiveness.
d = 2
Number of synonym candidates generated by the BERT MLM. Section 4.1 sets d=2; Section 4.5.2 shows d=3 further improves attack success, so the base setting is a hand-chosen compromise.

assumptions (5)

domain assumption Attention weights of a transformer reflect the importance of each input token for QA prediction.
Section 3.2 states that attention scores reveal the words most relevant to answering the question; this is the basis for ABR and is not proven.
standard math The difference in answer logits after deleting a token quantifies that token's importance.
Section 3.3, Eq. (1); this is the standard saliency measure used by prior deletion-based attacks such as TextFooler.
domain assumption BERT MLM can propose synonyms that preserve grammaticality and semantics in context.
Section 3.5 asserts this capability; the paper does not measure how often the replacement changes the intended meaning.
domain assumption Victim QA models expose attention weights and output logits.
Algorithm 1 requires an attention network A and logits L; many deployed QA services do not expose these internals.
ad hoc to paper topk=5 and d=2 are reasonable operating points for the base attack.
Section 4.1 sets these values; Section 4.5 shows higher topk and d increase attack success, so the choice trades off modification rate.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Deceiving Question-Answering Models: A Hybrid Word-Level Adversarial Approach." pith.science (2026). https://pith.science/paper/K4QSDT7V

@misc{pith2026241108248,
  author       = {Pith},
  title        = {Pith review of: Deceiving Question-Answering Models: A Hybrid Word-Level Adversarial Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K4QSDT7V}},
  note         = {Machine review of arXiv:2411.08248}
}

read the original abstract

Deep learning underpins most of the currently advanced natural language processing (NLP) tasks such as textual classification, neural machine translation (NMT), abstractive summarization and question-answering (QA). However, the robustness of the models, particularly QA models, against adversarial attacks is a critical concern that remains insufficiently explored. This paper introduces QA-Attack (Question Answering Attack), a novel word-level adversarial strategy that fools QA models. Our attention-based attack exploits the customized attention mechanism and deletion ranking strategy to identify and target specific words within contextual passages. It creates deceptive inputs by carefully choosing and substituting synonyms, preserving grammatical integrity while misleading the model to produce incorrect responses. Our approach demonstrates versatility across various question types, particularly when dealing with extensive long textual inputs. Extensive experiments on multiple benchmark datasets demonstrate that QA-Attack successfully deceives baseline QA models and surpasses existing adversarial techniques regarding success rate, semantics changes, BLEU score, fluency and grammar error rate.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 64 canonical work pages

[63]

In: Walker, M., Ji, H., Stent, A

Yasunaga, M., Kasai, J., Radev, D.: Robust multilingual part-of-speech tagging via adversarial training. In: Walker, M., Ji, H., Stent, A. (eds.) Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pp. 976–986. Association for Computationa...

work page 2018
[1]

Antaki, F., Milad, D., Chia, M.A., Gigu` ere, C. ´E., Touma, S., El-Khoury, J., Keane, P.A., Duval, R.: Capabilities of gpt-4 in ophthalmology: an analysis of model entropy and progress towards human-level medical question answering. British Journal of Ophthalmology 108(10), 1371–1378 (2024) 3Different sizes of BERT models can be obtained from https://git...

work page 2024
[2]

CoRR abs/1409.0473 (2014)

Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473 (2014)

arXiv 2014
[3]

Bongini, P., Becattini, F., Del Bimbo, A.: Is gpt-3 all you need for visual question answering in cultural heritage? In: European Conference on Computer Vision. pp. 268–281. Springer (2022)

work page 2022
[4]

In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A....

2020
[5]

In: Goldberg, Y., Kozareva, Z., Zhang, Y

Cao, Y., Li, D., Fang, M., Zhou, T., Gao, J., Zhan, Y., Tao, D.: TASA: Deceiv- ing question answering models by twin answer sentences attack. In: Goldberg, Y., Kozareva, Z., Zhang, Y. (eds.) Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing. pp. 11975–11992. Association for Computational Linguistics, Abu Dhabi, Unite...

work page 2022
[6]

In: Burstein, J., Doran, C., Solorio, T

Clark, C., Lee, K., Chang, M.W., Kwiatkowski, T., Collins, M., Toutanova, K.: BoolQ: Exploring the surprising difficulty of natural yes/no questions. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long ...

work page 2019
[7]

In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies

Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies. pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minne...

work page 2019

Show all 73 references

[8]

In: International conference on machine learning for cyber security

Dong, H., Dong, J., Yuan, S., Guan, Z.: Adversarial attack and defense on natural language processing in deep learning: A survey and perspective. In: International conference on machine learning for cyber security. pp. 409–424. Springer (2022)

2022
[9]

In: Inter- national conference on natural language processing and knowledge engineering,

Dong, Z., Dong, Q.: Hownet-a hybrid language and knowledge resource. In: Inter- national conference on natural language processing and knowledge engineering,
[10]

In: Gurevych, I., Miyao, Y

Ebrahimi, J., Rao, A., Lowd, D., Dou, D.: HotFlip: White-box adversarial exam- ples for text classification. In: Gurevych, I., Miyao, Y. (eds.) Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 31–36. Associat...

2018
[11]

IEEE Transactions on Neural Networks and Learning Systems 32(10), 4291–4308 (Oct 2021) 22

Galassi, A., Lippi, M., Torroni, P.: Attention in natural language processing. IEEE Transactions on Neural Networks and Learning Systems 32(10), 4291–4308 (Oct 2021) 22

2021
[12]

In: Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP)

Garg, S., Ramakrishnan, G.: Bae: Bert-based adversarial examples for text clas- sification. In: Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP). Association for Computational Linguistics (2020)

2020
[13]

ACM Computing Surveys 55(14s), 1–39 (2023)

Goyal, S., Doddapaneni, S., Khapra, M.M., Ravindran, B.: A survey of adversarial defenses and robustness in nlp. ACM Computing Surveys 55(14s), 1–39 (2023)

2023
[14]

In: Carpuat, M., de Marneffe, M.C., Meza Ruiz, I.V

Guo, M., Ainslie, J., Uthus, D., Ontanon, S., Ni, J., Sung, Y.H., Yang, Y.: LongT5: Efficient text-to-text transformer for long sequences. In: Carpuat, M., de Marneffe, M.C., Meza Ruiz, I.V. (eds.) Findings of the Association for Compu- tational Linguistics: NAACL 2022. pp. 72...

2022
[15]

Security and Privacy5(3), e209 (2022)

Hathaliya, J.J., Tanwar, S., Sharma, P.: Adversarial learning techniques for secu- rity and privacy preservation: A comprehensive review. Security and Privacy5(3), e209 (2022)

2022
[16]

In: Walker, M., Ji, H., Stent, A

Iyyer, M., Wieting, J., Gimpel, K., Zettlemoyer, L.: Adversarial example gener- ation with syntactically controlled paraphrase networks. In: Walker, M., Ji, H., Stent, A. (eds.) Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computati...

2018
[17]

In: Palmer, M., Hwa, R., Riedel, S

Jia, R., Liang, P.: Adversarial examples for evaluating reading comprehension systems. In: Palmer, M., Hwa, R., Riedel, S. (eds.) Proceedings of the 2017 Con- ference on Empirical Methods in Natural Language Processing. pp. 2021–2031. Association for Computational Linguistics,...

2017
[18]

Proceedings of the AAAI Conference on Artificial Intelligence 34(05), 8018–8025 (Apr 2020)

Jin, D., Jin, Z., Zhou, J.T., Szolovits, P.: Is bert really robust? a strong baseline for natural language attack on text classification and entailment. Proceedings of the AAAI Conference on Artificial Intelligence 34(05), 8018–8025 (Apr 2020)

2020
[19]

Applied Sciences 11(14), 6421 (2021)

Jin, D., Pan, E., Oufattole, N., Weng, W.H., Fang, H., Szolovits, P.: What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11(14), 6421 (2021)

2021
[20]

(eds.) Proceedings of the 22nd Conference on Computational Natural Language Learning

Kann, K., Rothe, S., Filippova, K.: Sentence-level fluency evaluation: References help, but can be spared! In: Korhonen, A., Titov, I. (eds.) Proceedings of the 22nd Conference on Computational Natural Language Learning. pp. 313–323. Association for Computational Linguistics, ...

2018
[21]

In: Cohn, T., He, Y., Liu, Y

Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., Hajishirzi, H.: UNIFIEDQA: Crossing format boundaries with a single QA system. In: Cohn, T., He, Y., Liu, Y. (eds.) Findings of the Association for Computational Linguis- tics: EMNLP 2020. pp. 1896–1907. A...

2020
[22]

arXiv preprint arXiv:1911.02365 (2019)

Klein, T., Nabi, M.: Learning to answer by learning to ask: Getting the best of gpt-2 and bert worlds. arXiv preprint arXiv:1911.02365 (2019)

2019 arXiv
[23]

In: Proceedings of the 2017 Conference on Designing Interactive Systems

Klopfenstein, L.C., Delpriori, S., Malatini, S., Bogliolo, A.: The rise of bots: A survey of conversational interfaces, patterns, and paradigms. In: Proceedings of the 2017 Conference on Designing Interactive Systems. p. 555–565. DIS ’17, Association for Computing Machinery, N...

2017
[24]

In: Webber, B., Cohn, T., He, Y., Liu, Y

Ko, M., Lee, J., Kim, H., Kim, G., Kang, J.: Look at the first sentence: Position bias in question answering. In: Webber, B., Cohn, T., He, Y., Liu, Y. (eds.) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 1109–1121. Associa...

2020
[25]

Transac- tions of the Association for Computational Linguistics 6, 317–328 (2018)

Koˇ cisk´ y, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K.M., Melis, G., Grefenstette, E.: The NarrativeQA reading comprehension challenge. Transac- tions of the Association for Computational Linguistics 6, 317–328 (2018)

2018
[26]

In: International Conference on Learning Representations (2020)

Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: Albert: A lite bert for self-supervised learning of language representations. In: International Conference on Learning Representations (2020)

2020
[27]

In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J

Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoy- anov, V., Zettlemoyer, L.: BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J. ...

2020
[28]

In: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cot- terell, R., Chakraborty, T., Zhou, Y

Li, D., Zhang, Y., Peng, H., Chen, L., Brockett, C., Sun, M.T., Dolan, B.: Contextualized perturbation for textual adversarial attack. In: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cot- terell, R., Chakraborty, T., Zhou, Y. (eds....

2021
[29]

In: Proceedings 2019 Network and Distributed System Security Symposium

Li, J., Ji, S., Du, T., Li, B., Wang, T.: Textbugger: Generating adversarial text against real-world applications. In: Proceedings 2019 Network and Distributed System Security Symposium. NDSS 2019, Internet Society (2019)

2019
[30]

In: Proceedings of the AAAI Conference on Artificial Intelligence

Li, J., Liu, W.: Summarization attack via paraphrasing (student abstract). In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 16250– 16251 (2023)

2023
[31]

In: Webber, B., Cohn, T., He, Y., Liu, Y

Li, L., Ma, R., Guo, Q., Xue, X., Qiu, X.: BERT-ATTACK: Adversarial attack against BERT using BERT. In: Webber, B., Cohn, T., He, Y., Liu, Y. (eds.) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 6193–6202. Association for C...

2020
[32]

In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence

Liang, B., Li, H., Su, M., Bian, P., Li, X., Shi, W.: Deep text classification can be fooled. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence. p. 4208–4215. International Joint Conferences on Artificial Intelligence Organization ...

2018
[33]

In: Text summarization branches out

Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out. pp. 74–81 (2004)

2004
[34]

In: M` arquez, L., Callison-Burch, C., Su, J

Luong, T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. In: M` arquez, L., Callison-Burch, C., Su, J. (eds.) Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. pp. 1412–1421. Association for C...

2015
[35]

Computer Science Review 48, 100552 (2023)

Martinez-Gil, J.: A survey on legal question–answering systems. Computer Science Review 48, 100552 (2023)

2023
[36]

In: Proceedings of the 1st International Conference on Learning Representations (ICLR) (2013)

Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word rep- resentations in vector space. In: Proceedings of the 1st International Conference on Learning Representations (ICLR) (2013)

2013
[37]

In: Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992 (1992)

Miller, G.A.: WordNet: A lexical database for English. In: Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992 (1992)

1992
[38]

In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume

Mozes, M., Stenetorp, P., Kleinberg, B., Griffin, L.: Frequency-guided word sub- stitutions for detecting textual adversarial examples. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume. pp. 171–186. ...

2021
[39]

GRIN Verlag (2003)

Naber, D.: A Rule-Based Style and Grammar Checker. GRIN Verlag (2003)

2003
[40]

In: 2018 IEEE 15th International Conference on e-Business Engineering (ICEBE)

Nuruzzaman, M., Hussain, O.K.: A survey on chatbot implementation in customer service industry through deep neural networks. In: 2018 IEEE 15th International Conference on e-Business Engineering (ICEBE). pp. 54–61 (2018)

2018
[41]

In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics

Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics. pp. 311–318 (2002)

2002
[42]

OpenAI blog 1(8), 9 (2019)

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)

2019
[43]

Journal of machine learning research 21(140), 1–67 (2020)

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21(140), 1–67 (2020)

2020
[44]

In: Gurevych, I., Miyao, Y

Rajpurkar, P., Jia, R., Liang, P.: Know what you don’t know: Unanswerable questions for SQuAD. In: Gurevych, I., Miyao, Y. (eds.) Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 2: Short Papers). pp. 784–789. Association for Co...

2018
[45]

In: Su, J., Duh, K., Carreras, X

Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ questions for machine comprehension of text. In: Su, J., Duh, K., Carreras, X. (eds.) Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pp. 2383–2392. Association for Comp...

2016
[46]

In: Korhonen, A., Traum, D., M` arquez, L

Ren, S., Deng, Y., He, K., Che, W.: Generating natural language adversarial examples through probability weighted word saliency. In: Korhonen, A., Traum, D., M` arquez, L. (eds.) Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 1085–...

2019
[47]

arXiv preprint arXiv:2104.07646 (2021) 25

Rosenthal, S., Bornea, M., Sil, A.: Are multilingual bert models robust? a case study on adversarial attacks for multilingual question answering. arXiv preprint arXiv:2104.07646 (2021) 25

2021 arXiv
[48]

In: Proceedings of the 5th Work- shop on Energy Efficient Machine Learning and Cognitive Computing

Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In: Proceedings of the 5th Work- shop on Energy Efficient Machine Learning and Cognitive Computing. NeurIPS, Vancouver, Canada (2019)

2019
[49]

In: Proceedings of the Workshop on Knowledge Augmented Methods for NLP (KnowledgeNLP) at AAAI 2023 (2023)

Shinoda, K., Sugawara, S., Aizawa, A.: Penalizing confident predictions on largely perturbed inputs does not improve out-of-distribution generalization in question answering. In: Proceedings of the Workshop on Knowledge Augmented Methods for NLP (KnowledgeNLP) at AAAI 2023 (2023)

2023
[50]

Journal of King Saud University-Computer and Information Sciences 32(6), 635–646 (2020)

Soares, M.A.C., Parreiras, F.S.: A literature review on question answering tech- niques, paradigms and systems. Journal of King Saud University-Computer and Information Sciences 32(6), 635–646 (2020)

2020
[51]

Advances in Neural Information Processing Systems 33, 3008–3021 (2020)

Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., Christiano, P.F.: Learning to summarize with human feedback. Advances in Neural Information Processing Systems 33, 3008–3021 (2020)

2020
[52]

IEEE Transactions on Knowledge and Data Engineering 35(4), 3367–3388 (2021)

Sun, H., Zhu, T., Zhang, Z., Jin, D., Xiong, P., Zhou, W.: Adversarial attacks against deep generative models on data: a survey. IEEE Transactions on Knowledge and Data Engineering 35(4), 3367–3388 (2021)

2021
[53]

In: Korhonen, A., Traum, D., M` arquez, L

Talmor, A., Berant, J.: MultiQA: An empirical investigation of generalization and transfer in reading comprehension. In: Korhonen, A., Traum, D., M` arquez, L. (eds.) Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics. pp. 4911–4921. Asso...

2019
[54]

In: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1

Toutanova, K., Klein, D., Manning, C.D., Singer, Y.: Feature-rich part-of-speech tagging with a cyclic dependency network. In: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volum...

2003
[55]

In: Blunsom, P., Bordes, A., Cho, K., Cohen, S., Dyer, C., Grefenstette, E., Hermann, K.M., Rimell, L., Weston, J., Yih, S

Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., Sule- man, K.: NewsQA: A machine comprehension dataset. In: Blunsom, P., Bordes, A., Cho, K., Cohen, S., Dyer, C., Grefenstette, E., Hermann, K.M., Rimell, L., Weston, J., Yih, S. (eds.) Proceedings of th...

2017
[56]

In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)

Wallace, E., Feng, S., Kandpal, N., Gardner, M., Singh, S.: Universal adversarial triggers for attacking and analyzing NLP. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language...

2019
[57]

Transactions of the Association for Computational Linguistics 7, 387–401 (2019)

Wallace, E., Rodriguez, P., Feng, S., Yamada, I., Boyd-Graber, J.: Trick me if you can: Human-in-the-loop generation of adversarial examples for question answer- ing. Transactions of the Association for Computational Linguistics 7, 387–401 (2019)

2019
[58]

In: Webber, B., Cohn, 26 T., He, Y., Liu, Y

Wang, B., Pei, H., Pan, B., Chen, Q., Wang, S., Li, B.: T3: Tree-autoencoder con- strained adversarial text generation for targeted attack. In: Webber, B., Cohn, 26 T., He, Y., Liu, Y. (eds.) Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Process...

2020
[59]

arXiv preprint arXiv:2206.15030 (2022)

Wang, Z.: Modern question answering datasets and benchmarks: A survey. arXiv preprint arXiv:2206.15030 (2022)

2022 arXiv
[60]

Xiao, T., Zhu, J.: Introduction to transformers: an nlp perspective (2023)

2023
[61]

In: International conference on machine learning

Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. In: International conference on machine learning. pp. 2048–2057. PMLR (2015)

2015
[62]

IEEE Internet of Things Journal 8(6), 4980–4990 (2020)

Yang, X., Liu, W., Zhang, S., Liu, W., Tao, D.: Targeted attention attack on deep learning models in road sign recognition. IEEE Internet of Things Journal 8(6), 4980–4990 (2020)

2020
[64]

Knowledge and Information Systems 66, 7165–7204 (2024)

Yigit, G., Amasyali, M.F.: From text to multimodal: A comprehensive survey of adversarial example generation in question answering systems. Knowledge and Information Systems 66, 7165–7204 (2024)

2024
[65]

In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J

Yu, J., Liu, W., Qiu, S., Su, Q., Wang, K., Quan, X., Yin, J.: Low-resource generation of multi-hop reasoning questions. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J. (eds.) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. ...

2020
[66]

In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J

Zang, Y., Qi, F., Yang, C., Liu, Z., Zhang, M., Liu, Q., Sun, M.: Word-level tex- tual adversarial attacking as combinatorial optimization. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J. (eds.) Proceedings of the 58th Annual Meeting of the Association for Computationa...

2020
[67]

Computational Linguistics 49(2), 395–427 (jun 2023)

Zeng, J., Zheng, X., Xu, J., Li, L., Yuan, L., Huang, X.: Certified robustness to text adversarial attacks by randomized [mask]. Computational Linguistics 49(2), 395–427 (jun 2023)

2023
[68]

In: International conference on machine learning

Zhang, J., Zhao, Y., Saleh, M., Liu, P.: Pegasus: Pre-training with extracted gap- sentences for abstractive summarization. In: International conference on machine learning. pp. 11328–11339. PMLR (2020)

2020
[69]

arXiv preprint arXiv:2101.00774 (2021)

Zhu, F., Lei, W., Wang, C., Zheng, J., Poria, S., Chua, T.S.: Retrieving and read- ing: A comprehensive survey on open-domain question answering. arXiv preprint arXiv:2101.00774 (2021)

2021 arXiv
[70]

In: Li, S., Sun, M., Liu, Y., Wu, H., Liu, K., Che, W., He, S., Rao, G

Zhuang, L., Wayne, L., Ya, S., Jun, Z.: A robustly optimized BERT pre-training approach with post-training. In: Li, S., Sun, M., Liu, Y., Wu, H., Liu, K., Che, W., He, S., Rao, G. (eds.) Proceedings of the 20th Chinese National Conference on Computational Linguistics. pp. 1218...

2021
[2003]

Proceedings. 2003. pp. 820–824. IEEE (2003)

2003
[2162]

Association for Computational Linguistics, Hong Kong, China (November 2019)

2019
[2936]

Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019)

2019

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [63]

In: Walker, M., Ji, H., Stent, A

Yasunaga, M., Kasai, J., Radev, D.: Robust multilingual part-of-speech tagging via adversarial training. In: Walker, M., Ji, H., Stent, A. (eds.) Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pp. 976–986. Association for Computationa...

work page 2018

[2] [1]

Antaki, F., Milad, D., Chia, M.A., Gigu` ere, C. ´E., Touma, S., El-Khoury, J., Keane, P.A., Duval, R.: Capabilities of gpt-4 in ophthalmology: an analysis of model entropy and progress towards human-level medical question answering. British Journal of Ophthalmology 108(10), 1371–1378 (2024) 3Different sizes of BERT models can be obtained from https://git...

work page 2024

[3] [2]

CoRR abs/1409.0473 (2014)

Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473 (2014)

arXiv 2014

[4] [3]

Bongini, P., Becattini, F., Del Bimbo, A.: Is gpt-3 all you need for visual question answering in cultural heritage? In: European Conference on Computer Vision. pp. 268–281. Springer (2022)

work page 2022

[5] [4]

In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A....

2020

[6] [5]

In: Goldberg, Y., Kozareva, Z., Zhang, Y

Cao, Y., Li, D., Fang, M., Zhou, T., Gao, J., Zhan, Y., Tao, D.: TASA: Deceiv- ing question answering models by twin answer sentences attack. In: Goldberg, Y., Kozareva, Z., Zhang, Y. (eds.) Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing. pp. 11975–11992. Association for Computational Linguistics, Abu Dhabi, Unite...

work page 2022

[7] [6]

In: Burstein, J., Doran, C., Solorio, T

Clark, C., Lee, K., Chang, M.W., Kwiatkowski, T., Collins, M., Toutanova, K.: BoolQ: Exploring the surprising difficulty of natural yes/no questions. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long ...

work page 2019

[8] [7]

In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies

Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies. pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minne...

work page 2019