Pith. sign in

REVIEW 4 major objections 4 minor 73 references

Confidence Elicitation: A New Attack Vector for Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Asking a large language model to state its own confidence yields a usable black-box signal: greedily substituting words that lower the elicited confidence drives the model into misclassification, outperforming hard-label-only attacks.

desk verdict Confidence elicitation as an attack signal is a genuinely new and well-supported idea; the main gap is that calibration is only shown on clean inputs, not on the perturbed search path. read the letter →

arxiv 2502.04643 v2 pith:3KLY6TET submitted 2025-02-07 cs.LG cs.CLcs.CR

Brian Formento , Chuan Sheng Foo , See-Kiong Ng This is my paper · ORCID
classification cs.LGcs.CLcs.CR
keywords adversarialattacksconfidenceelicitationlargelanguagemodelsblack-boxhard-labelwordsubstitutioncalibrationtextclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models reached through APIs expose nothing but their generated text, so black-box attacks have had to search using only the final label as feedback. The paper claims that the model itself volunteers more: prompted for its guesses and its confidence in them, an LLM gives a verbalised self-assessment that behaves like a calibrated probability, and minimising that number through word substitutions raises the chance of a label flip. The authors implement this as CEAttack — query for $k$ guesses plus a verbal confidence for each, blend them through a Dirichlet mean, and keep any synonym substitution that lowers the mean for the current class — and report attack success rates of roughly 18–46%, above three hard-label baselines, on two 7–8 billion parameter models and three datasets. Confidence elicitation is being promoted as a reliability feature for high-stakes deployment; if the paper is right, that same feature is also an attack surface, and the threat model for closed-model APIs is wider than previously assumed.

What carries the argument

The load-bearing mechanism is the Dirichlet aggregator applied to elicited verbal confidence. The attack first asks the model for $k$ guesses for the input's class, then asks for a verbal confidence attached to each guess ('Highest' = 5, 'High' = 4, 'Medium' = 3, 'Low' = 2, 'Lowest' = 1); these values populate a per-class $\alpha$-vector, and the Dirichlet mean $\mu_c = \alpha_c / \sum_j \alpha_j$ is treated as the approximate probability of class $c$. That single scalar is what converts a hard-label query into something functionally close to a soft label. The second piece is the greedy word-substitution loop: words are drawn from an ordered subset capped at $|W| = 5$, each is replaced in turn by synonyms from counter-fitted embeddings, and a substitution is accepted only if it lowers $\mu_y$ for the current class; the search continues until the argmax class flips, and the confidence drop is the feedback that hard-label-only attacks lack.

What would settle it

Measure the AUROC or ECE of the elicited Dirichlet-mean confidence on the perturbed samples the search actually accepts, rather than on clean inputs; if the signal fails to separate correct from incorrect predictions there (AUROC near $0.5$) while the attack still flips labels, the claim that minimising confidence drives misclassification would be refuted. A matched-query-budget control — random synonym substitutions instead of confidence-guided ones — that ties CEAttack's success rate would likewise show the confidence signal adds nothing.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that verbalised confidence is a standing attack vector: in a fully black-box classification setting, the confidence a model states for its answer can stand in for the output probabilities that soft-label attacks use. The authors formalise the intuition as $\rho(\theta) = -\mathbb{E}_{(x,y)\sim\mathcal{D}}[\max_{\delta\in\Delta} C(\theta, x+\delta, y)]$: minimising the confidence $C$ in the correct class is linked to maximising the probability of misclassification. Concretely, CEAttack prompts the model twice — first for its $k$ most likely class guesses, then for a verbal confidence ('Highest' to 'Lowest') attached to each guess — maps the responses to a Dirichlet distribution over classes, and treats the Dirichlet mean as the approximate probability of the current class. A greedy hill-climbing loop then tries synonym substitutions from counter-fitted embeddings and commits to any substitution that lowers that mean, stopping when the predicted class changes. The paper reports this achieves state-of-the-art attack success among hard-label, black-box, word-substitution attacks on the models and datasets tested, with comparable or better semantic similarity and fewer queries for successful attacks than its principal hard-label baseline, and shows in the appendix that the same mechanism degrades a closed-source API model's accuracy as well.

Load-bearing premise

The attack rests on the assumption that the confidence a model states for its own answer, once blended through the paper's chosen Dirichlet weights, keeps tracking how likely that answer is to be correct on the tampered sentences the search generates, but the paper checks calibration only on clean inputs — where the signal is reasonably calibrated on two datasets and close to chance on the third (StrategyQA, AUROC $0.66$).

Editorial extensions

If this is right

  • Hard-label word-substitution attacks no longer have to over-perturb first and repair later: a bottom-up greedy path guided by confidence reaches a label flip with fewer queries, which matters when APIs are rate-limited.
  • Any interface that returns a model's self-assessed confidence — a feature marketed as transparency or reliability — is also handing attackers a continuous optimisation signal, since the whole loop uses nothing but the generated text.
  • The authors argue the mechanism is agnostic to the perturbation type, the search algorithm, and the quality bound, so the same feedback should transfer to character-level edits, sentence-level rewrites, and multi-turn jailbreak prompts.
  • Better calibration makes the attack stronger: as models improve at expressing uncertainty — an emergent ability at larger scale — the minimised confidence becomes a more trustworthy proxy, coupling progress in calibration with progress in attack success.
  • The appendix shows the same mechanism degrades a closed-source API model on all three datasets, indicating the vector is generic rather than an artefact of the two 7–8 billion parameter models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper validates calibration only on clean inputs; a direct next experiment is to measure whether the Dirichlet-mean signal remains informative on the perturbed inputs the search accepts, since attack success would plateau if the signal degrades there.
  • The confidence weights ('Highest' = 5 through 'Lowest' = 1) are hand-chosen; tuning them per model, or comparing the Dirichlet mean against the raw guess frequencies without confidence weighting, would isolate how much guidance comes from verbal confidence versus the guess distribution itself — the paper compares self-consistency only for calibration, not for attack guidance.
  • If the mechanism generalises beyond fixed-class classification, a calibrated assistant that reports uncertainty on every open-ended answer gives an attacker a continuous target for jailbreak-style optimisation; the authors flag this direction but do not test it, and the 'defense by intent' rules they sketch (repeated near-identical queries, explicit confidence requests, confidence drift over turns
  • The defensive options the paper sketches — adding noise to confidence, deliberately degrading calibration — would sacrifice the reliability that motivates confidence elicitation in high-stakes deployment, so the viable responses may be architectural or behavioural rather than suppression of confidence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces CEAttack, a black-box hard-label attack on LLM text classification that uses elicited verbal confidence as an approximate soft-label feedback signal. The attack first queries the model for k guesses and verbal confidence levels, aggregates them via a hand-chosen Dirichlet distribution to obtain an empirical mean probability for each class, and then performs greedy word-substitution guided by minimizing the mean of the currently predicted class. The authors evaluate the method on SST-2, AG-News, and StrategyQA with two instruction-tuned LLMs, reporting higher attack success rates than Self-Fool Word Sub, TextHoaxer, and SSPAttack, with comparable semantic similarity and lower query counts on successful attacks. They also provide calibration metrics (ECE, AUROC) for the elicited confidence on clean inputs, a temperature ablation, and qualitative attack-path examples.

Significance. If the central mechanism holds, the paper demonstrates a genuinely new attack vector: closed-source LLMs that expose confidence elicitation are substantially more vulnerable to black-box word-substitution attacks than previously known, and this soft-label-like information can be extracted even when logits are unavailable. The idea is well-motivated, the experiments span multiple datasets and models, and the authors release their code, which supports reproducibility. However, the significance rests on the assumption that the elicited confidence remains a reliable proxy for the probability of misclassification on the perturbed candidates the search actually visits; the current evidence for that is only indirect and, on StrategyQA, the clean-input AUROC is close to chance. The work is therefore promising but needs stronger validation before the state-of-the-art claim can be fully accepted.

major comments (4)
  1. [Section 5.1 and Algorithm 1, Eq. (3)] The calibration evidence (Table 1, Figure 3) is computed exclusively on clean, unperturbed inputs, whereas the attack's acceptance rule (Algorithm 1, line 7) uses a drop in the Dirichlet mean on perturbed candidates to decide whether to retain a substitution. The paper never reports ECE or AUROC on the distribution of perturbed samples that the greedy search actually evaluates, nor does it test whether a confidence drop along the search path predicts a subsequent label flip. This gap is particularly acute for StrategyQA, where the clean AUROC is only 0.66 (LLaMA-3) and 0.64 (Mistral) and where CEAttack shows its largest absolute ASR improvement over SSPAttack. Without perturbed-distribution calibration, the higher ASR could be produced by the iterative word-substitution search itself rather than by the confidence signal; the comparison to the non-iterative Self-Fool Word Sub baseline does not control for the number of re-queries, and the H.5 ablation shows only that the specific Dirichlet aggregator outperforms a simpler numerical confidence signal, not that the signal is reliable under perturbation.
  2. [Section 5.2, Tables 2–4] The main attack results are reported as point estimates without error bars, confidence intervals, or significance tests. Since the set of candidate words W is selected randomly (Section 4.1, 'we first extract an ordered subset of words W ... randomly'), the ASR, AUA, SemSim, and query counts should exhibit run-to-run variance. Without multiple seeds or a statistical comparison, the claimed state-of-the-art advantage over SSPAttack and TextHoaxer on specific datasets is not rigorously established. At minimum, the authors should report variance over at least three random word-selection seeds or justify determinism of the full pipeline.
  3. [Section 4.1, Dirichlet aggregation] The mapping from verbal confidence labels to Dirichlet pseudo-counts ('Highest'=5, 'High'=4, ..., 'Lowest'=1) and the choice of α0=1 are presented as a fait accompli with no sensitivity analysis or justification that this particular aggregation approximates the model's class probability distribution. The H.5 ablation (Table 18) shows that replacing this aggregator with a simple k=1 numerical confidence reduces ASR, but that does not validate the Dirichlet mean as a calibrated probability estimate on the perturbed samples used during search. Because Eq. (3) minimizes exactly this mean, the entire mechanism depends on the empirical mean being monotonically informative about the chance of misclassification under perturbation; the paper should provide perturbed-input calibration or a correlation analysis between Δμ and label flips.
  4. [Appendix H.3, Tables 14–16] The temperature ablation reveals substantial sensitivity to the decoding temperature: for LLaMA-3 on SST-2, CEAttack's ASR rises from 19.73 at τ≈0 to 35.79 at τ=0.7, while SSPAttack's ASR drops from 9.73 to 5.42. The main text presents only the τ≈0 results as the primary comparison and does not discuss this temperature dependence. If the attack's effectiveness is highly sensitive to temperature, the state-of-the-art claim is contingent on a configuration detail that is not highlighted in the main body; the authors should either present both settings in the main results or clearly explain why τ≈0 is the appropriate operational regime for their threat model.
minor comments (4)
  1. [Section 4.1 and Table 6] The description of the two-step prompting strategy is slightly unclear in the main text; Table 6 shows that the second prompt refers to the 'previous prompt' and the generated guesses, but the exact formatting of the confidence output (e.g., how 'Highest' is mapped to a number) is not stated in the main body. A brief example of the full prompt and parsed output would aid reproducibility.
  2. [Tables 2 and 14] The ASR values for CEAttack at τ≈0 and τ=0.7 differ dramatically (e.g., SST2/LLaMA: 19.73 vs 35.79; AG-News/Mistral: 38.33 vs 78.44). The main text does not mention this sensitivity, and readers may mistakenly interpret the τ≈0 numbers as representative of the method's peak performance. Please add a sentence in Section 5.2 or Appendix H.3 that explicitly addresses this.
  3. [Section 5.2 and Table 3] The semantic similarity values are reported to two decimals, but the text states that 'successful perturbations in our study all have an angular semantic similarity of at least ε=0.84'; it would be helpful to report the distribution of SemSim (e.g., minimum, percentile) rather than only the mean, especially since the qualitative examples in Appendix J sometimes show substitutions that appear semantically distant (e.g., 'appropriate' for 'good enough').
  4. [Throughout] The paper alternates between 'LLaMa' and 'LLaMA' in text and tables; please standardize to 'LLaMA'. Also, the reference to 'Portillo Wightman et al., 2023' should include all authors (Portillo Wightman, Delucia, Dredze) for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the attack is an empirical score-guided search, and the paper's claims are tested against external ground-truth labels rather than derived from the confidence signal by definition.

full rationale

The paper's central link, stated in Equation (2), is that minimizing a calibrated confidence estimate for the correct class should increase the probability of misclassification. This is not circular: the confidence values are elicited verbal estimates aggregated by a fixed, hand-chosen Dirichlet weighting scheme ('Highest' = 5, 'High' = 4, etc.), not parameters fitted to attack outcomes. The ablation in Appendix H.5, where a simpler aggregator performs worse, confirms that the specific Dirichlet aggregation is an empirical design choice rather than a definitional trick. It is true that the paper defines the classifier's prediction as the argmax of the same Dirichlet mean used as the optimization signal, so lowering the score of the current class is a direct way to move the constructed decision boundary. However, the paper's headline claims are not forced by this construction: attack success rate is measured against external ground-truth labels, and the empirical content is that synonym substitutions found by the greedy search actually lower the elicited confidence and eventually flip the argmax (Tables 2 and 5). The clean-data calibration results (ECE, AUROC) are reported as empirical measurements, not as consequences of the definitions, and the lack of calibration evaluation on perturbed inputs is an evidence gap about generalization, not a circular reduction of the conclusion to its inputs. The paper contains no load-bearing self-citations and imports no uniqueness theorem; the authors' prior work appears only in related-work and future-work contexts. Overall, the derivation chain is not equivalent to its inputs by construction, so no circular step is identified.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim rests on an empirical bridge between self-reported confidence and true misclassification risk. That bridge is plausible and partially supported, but it is not derived, and its components (Dirichlet weights, prior, number of guesses) are chosen by hand. The paper tests the bridge with clean-data calibration, which is the right check, but leaves the perturbed-input regime unexamined.

free parameters (5)
  • Dirichlet confidence weights = Highest=5, High=4, Medium=3, Low=2, Lowest=1
    Hand-chosen mapping from verbal confidence levels to Dirichlet alpha contributions in Section 4.1. Not fitted to data, but arbitrary; changing them changes the estimated probabilities and therefore the attack path.
  • Dirichlet prior alpha_0 = 1
    Added to every class as a smoothing prior in the Dirichlet aggregation (Section 4.1). Arbitrary and not motivated by derivation.
  • Number of guesses k = 20 for SST-2 and AG-News, 6 for StrategyQA
    Chosen based on prior work or compute budget; affects the granularity of the confidence estimate.
  • Word perturbation budget |W| = 5
    Maximum number of words considered for substitution per sample (Section 4.1). Ablations show performance increases with |W|, so the default is a conservative choice.
  • Semantic similarity threshold epsilon = 0.84
    Used to filter low-quality perturbations; taken from prior work (Jin et al. 2019), not fitted here.
assumptions (3)
  • domain assumption Verbalized confidence reflects the model's true uncertainty about its predictions.
    Core assumption of the attack; tested in Section 5.1 via ECE and AUROC, with mixed results (e.g., StrategyQA AUROC 0.66).
  • ad hoc to paper The Dirichlet-weighted histogram of guesses and verbal confidences approximates the model's class probability distribution.
    The aggregation in Section 4.1 is a hand-designed recipe, not derived from a generative model of the LLM's outputs.
  • domain assumption Counter-fitted synonym substitutions preserve the original meaning well enough to be considered valid adversarial perturbations.
    Standard assumption in word-substitution attacks; the paper uses SemSim and perplexity to monitor quality but does not verify label preservation semantically.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Confidence Elicitation: A New Attack Vector for Large Language Models." pith.science (2026). https://pith.science/paper/3KLY6TET

@misc{pith2026250204643,
  author       = {Pith},
  title        = {Pith review of: Confidence Elicitation: A New Attack Vector for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3KLY6TET}},
  note         = {Machine review of arXiv:2502.04643}
}
read the original abstract

A fundamental issue in deep learning has been adversarial robustness. As these systems have scaled, such issues have persisted. Currently, large language models (LLMs) with billions of parameters suffer from adversarial attacks just like their earlier, smaller counterparts. However, the threat models have changed. Previously, having gray-box access, where input embeddings or output logits/probabilities were visible to the user, might have been reasonable. However, with the introduction of closed-source models, no information about the model is available apart from the generated output. This means that current black-box attacks can only utilize the final prediction to detect if an attack is successful. In this work, we investigate and demonstrate the potential of attack guidance, akin to using output probabilities, while having only black-box access in a classification setting. This is achieved through the ability to elicit confidence from the model. We empirically show that the elicited confidence is calibrated and not hallucinated for current LLMs. By minimizing the elicited confidence, we can therefore increase the likelihood of misclassification. Our new proposed paradigm demonstrates promising state-of-the-art results on three datasets across two models (LLaMA-3-8B-Instruct and Mistral-7B-Instruct-V0.3) when comparing our technique to existing hard-label black-box attack methods that introduce word-level substitutions.

Figures

Figures reproduced from arXiv: 2502.04643 by the authors.

Figure 1
Figure 1. Confidence elicitation attack on an LLM, assuming a classification task (Start), [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Confidence Elicitation Attack on an LLM: Left) SSPAttack and other previous hard [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Reliability plots. Top) We show the SST2, AG-News and StrategyQA on LLama 3 8B [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Ablation study on |W| and |S| Because CEAttack’s optimization path is more direct compared to SSPAttack as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Ternary plots highlighting the attack path for the example in Table 5. As the algorithm adds [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Reliability plots. On the top, we show the SST2, AG-News and StrategyQA on Mistralv0.3 [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 27 canonical work pages

  1. [1]

    Neeladri Bhuiya, Viktor Schlegel, and Stefan Winkler. Seemingly plausible distractors in multi-hop reasoning: Are large language models attentive readers? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 2514--2528, Miami, Florida, USA, November 2024. ...

    doi:10.18653/v1/2024.emnlp-main.147 2024

  2. [2]

    Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt

    Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ ...

    work page 2023

  3. [3]

    Universal sentence encoder

    Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. Universal sentence encoder. arXiv preprint arXiv:1803.11175, 2018

    arXiv 2018

  4. [4]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries, 2023

    2023

  5. [5]

    Finetuning language models to emit linguistic expressions of uncertainty, 2024

    Arslan Chaudhry, Sridhar Thiagarajan, and Dilan Gorur. Finetuning language models to emit linguistic expressions of uncertainty, 2024. URL https://arxiv.org/abs/2409.12180

    arXiv 2024

  6. [6]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pp.\ 4171--4186, Minneapo...

    doi:10.18653/v1/n19-1423 2019

  7. [7]

    Towards robustness against natural language word substitutions

    Xinshuai Dong, Anh Tuan Luu, Rongrong Ji, and Hong Liu. Towards robustness against natural language word substitutions. In International Conference on Learning Representations, 2021 a . URL https://openreview.net/forum?id=ks5nebunVn_

    work page 2021

  8. [8]

    Towards Robustness Against Natural Language Word Substitutions

    Xinshuai Dong, Anh Tuan Luu, Rongrong Ji, and Hong Liu. Towards robustness against natural language word substitutions, 2021 b . URL https://arxiv.org/abs/2107.13541

    work page Pith review arXiv 2021

Show all 73 references
  1. [9]

    H ot F lip: White-box adversarial examples for text classification

    Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. H ot F lip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp.\ 31--36, Melbourne, Australia, Ju...

    2018 doi

  2. [10]

    o zde G \

    Steffen Eger, G \"o zde G \"u l S ahin, Andreas R \"u ckl \'e , Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. Text processing like humans do: Visually attacking and shielding NLP systems. 2019

    2019

  3. [11]

    Special symbol attacks on nlp systems

    Brian Formento, See-Kiong Ng, and Chuan-Sheng Foo. Special symbol attacks on nlp systems. In 2021 International Joint Conference on Neural Networks (IJCNN), pp.\ 1--8, 2021. doi:10.1109/IJCNN52387.2021.9534254

    2021

  4. [12]

    Using punctuation as an adversarial attack on deep learning-based NLP systems: An empirical study

    Brian Formento, Chuan Sheng Foo, Luu Anh Tuan, and See Kiong Ng. Using punctuation as an adversarial attack on deep learning-based NLP systems: An empirical study. In Andreas Vlachos and Isabelle Augenstein (eds.), Findings of the Association for Computational Linguistics: EAC...

    2023 doi

  5. [13]

    S em R o D e: Macro adversarial training to learn representations that are robust to word-level attacks

    Brian Formento, Wenjie Feng, Chuan-Sheng Foo, Anh Tuan Luu, and See-Kiong Ng. S em R o D e: Macro adversarial training to learn representations that are robust to word-level attacks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of t...

    2024

  6. [14]

    Reasoning robustness of LLM s to adversarial typographical errors

    Esther Gan, Yiran Zhao, Liying Cheng, Mao Yancan, Anirudh Goyal, Kenji Kawaguchi, Min-Yen Kan, and Michael Shieh. Reasoning robustness of LLM s to adversarial typographical errors. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference ...

    2024 doi

  7. [15]

    Improving the robustness of question answering systems to question paraphrasing

    Wee Chung Gan and Hwee Tou Ng. Improving the robustness of question answering systems to question paraphrasing. In Anna Korhonen, David Traum, and Llu \' s M \`a rquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 6065--6...

    2019 doi

  8. [16]

    Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies . Transactions of the Association for Computational Linguistics, 9: 0 346--361, 04 2021. ISSN 2307-3...

    2021 doi

  9. [17]

    Buelow, Rupert Langer, Bastian Dislich, Peter Boor, Volkmar Schulz, and Jakob Nikolas Kather

    Narmin Ghaffari Laleh, Daniel Truhn, Gregory Patrick Veldhuizen, Tianyu Han, Marko van Treeck, Roman D. Buelow, Rupert Langer, Bastian Dislich, Peter Boor, Volkmar Schulz, and Jakob Nikolas Kather. Adversarial attacks and adversarial robustness in computational pathology. Natu...

    2022

  10. [18]

    Goodfellow, Jonathon Shlens, and Christian Szegedy

    Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2014. URL https://arxiv.org/abs/1412.6572

    2014 arXiv

  11. [19]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks, 2017 a

    2017

  12. [20]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.\ 1321-...

    2017

  13. [21]

    Catastrophic jailbreak of open-source llms via exploiting generation, 2023

    Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source llms via exploiting generation, 2023. URL https://arxiv.org/abs/2310.06987

    2023 arXiv

  14. [22]

    Adversarial example generation with syntactically controlled paraphrase networks

    Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North A merican Chapter of the Associ...

    2018 doi

  15. [23]

    Advlora: Adversarial low-rank adaptation of vision-language models

    Yuheng Ji, Yue Liu, Zhicheng Zhang, Zhao Zhang, Yuting Zhao, Gang Zhou, Xingwei Zhang, Xinwang Liu, and Xiaolong Zheng. Advlora: Adversarial low-rank adaptation of vision-language models. arXiv preprint arXiv:2404.13425, 2024

    2024 arXiv

  16. [24]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023

  17. [25]

    How can we know when language models know? on the calibration of language models for question answering, 2021

    Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering, 2021

    2021

  18. [26]

    Is bert really robust? a strong baseline for natural language attack on text classification and entailment, 2019

    Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment, 2019. URL https://arxiv.org/abs/1907.11932

    2019 arXiv

  19. [27]

    T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Lin...

    2017 doi

  20. [28]

    Language models (mostly) know what they know, 2022

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav For...

    2022

  21. [29]

    Adversarial examples in the physical world

    A Kurakin, I Goodfellow, and S Bengio. Adversarial examples in the physical world. arxiv 2016. arXiv preprint arXiv:1607.02533, 2016

    2016 arXiv

  22. [30]

    TextBugger : Generating adversarial text against real-world applications

    Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. TextBugger : Generating adversarial text against real-world applications. In Proceedings 2019 Network and Distributed System Security Symposium. Internet Society, 2019. doi:10.14722/ndss.2019.23138. URL https://doi.org/10.14722

    2019

  23. [31]

    BERT - ATTACK : Adversarial attack against BERT using BERT

    Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. BERT - ATTACK : Adversarial attack against BERT using BERT . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 6193--6202, Online, November 2020. Associatio...

    2020 doi

  24. [32]

    Teaching models to express their uncertainty in words, 2022

    Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words, 2022

    2022

  25. [33]

    Sspattack: A simple and sweet paradigm for black-box hard-label textual adversarial attack

    Han Liu, Zhi Xu, Xiaotong Zhang, Xiaoming Xu, Feng Zhang, Fenglong Ma, Hongyang Chen, Hong Yu, and Xianchao Zhang. Sspattack: A simple and sweet paradigm for black-box hard-label textual adversarial attack. Proceedings of the AAAI Conference on Artificial Intelligence, 37 0 (1...

    2023 doi

  26. [34]

    Uncertainty estimation and quantification for llms: A simple supervised approach, 2024 a

    Linyu Liu, Yu Pan, Xiaocheng Li, and Guanting Chen. Uncertainty estimation and quantification for llms: A simple supervised approach, 2024 a . URL https://arxiv.org/abs/2404.15993

    2024 arXiv

  27. [35]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models, 2024 b

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models, 2024 b

    2024

  28. [36]

    Flipattack: Jailbreak llms via flipping

    Yue Liu, Xiaoxin He, Miao Xiong, Jinlan Fu, Shumin Deng, and Bryan Hooi. Flipattack: Jailbreak llms via flipping. arXiv preprint arXiv:2410.02832, 2024 c

    2024 arXiv

  29. [37]

    At which training stage does code data help LLM s reasoning? In The Twelfth International Conference on Learning Representations, 2024

    Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help LLM s reasoning? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=KIPJKST4gw

    2024

  30. [38]

    Towards deep learning models resistant to adversarial attacks, 2019

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2019

    2019

  31. [40]

    Generating natural language attacks in a hard label black box setting

    Rishabh Maheshwary, Saket Maheshwary, and Vikram Pudi. Generating natural language attacks in a hard label black box setting. In AAAI Conference on Artificial Intelligence, 2020 b . doi:10.48550/ARXIV.2012.14956

    2020 doi arXiv

  32. [41]

    Tree of attacks: Jailbreaking black-box LLM s automatically

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum S Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box LLM s automatically. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://o...

    2024

  33. [42]

    Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi

    John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp, 2020. URL https://arxiv.org/abs/2005.05909

    2020 arXiv

  34. [43]

    Counter-fitting word vectors to linguistic constraints, 2016

    Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Lina Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. Counter-fitting word vectors to linguistic constraints, 2016

    2016

  35. [44]

    Strength in numbers: Estimating confidence of large language models by prompt agreement

    Gwenyth Portillo Wightman, Alexandra Delucia, and Mark Dredze. Strength in numbers: Estimating confidence of large language models by prompt agreement. In Anaelia Ovalle, Kai-Wei Chang, Ninareh Mehrabi, Yada Pruksachatkun, Aram Galystan, Jwala Dhamala, Apurv Verma, Trista Cao,...

    2023

  36. [45]

    Extreme miscalibration and the illusion of adversarial robustness

    Vyas Raina, Samson Tan, Volkan Cevher, Aditya Rawal, Sheng Zha, and George Karypis. Extreme miscalibration and the illusion of adversarial robustness. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa...

    2024 doi

  37. [46]

    Generating natural language adversarial examples through probability weighted word saliency

    Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 1085--1097, Florence, Italy, July 201...

    2019 doi

  38. [47]

    Great, now write an article about that: The crescendo multi-turn llm jailbreak attack, 2024

    Mark Russinovich, Ahmed Salem, and Ronen Eldan. Great, now write an article about that: The crescendo multi-turn llm jailbreak attack, 2024. URL https://arxiv.org/abs/2404.01833

    2024 arXiv

  39. [48]

    Second-order uncertainty quantification: A distance-based approach

    Yusuf Sale, Viktor Bengs, Michele Caprio, and Eyke H\" u llermeier. Second-order uncertainty quantification: A distance-based approach. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedi...

    2024

  40. [49]

    Large language model uncertainty measurement and calibration for medical diagnosis and treatment

    Thomas Savage, John Wang, Robert Gallo, Abdessalem Boukil, Vishwesh Patel, Seyed Amir Ahmad Safavi-Naini, Ali Soroush, and Jonathan H Chen. Large language model uncertainty measurement and calibration for medical diagnosis and treatment. medRxiv, 2024. doi:10.1101/2024.06.06.24308399

    2024 doi

  41. [50]

    Logan IV, Eric Wallace, and Sameer Singh

    Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. A uto P rompt: E liciting K nowledge from L anguage M odels with A utomatically G enerated P rompts. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conferen...

    2020 doi

  42. [51]

    Goodfellow, and Rob Fergus

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, ...

    2014 arXiv

  43. [52]

    It’s morphin’ time! combating linguistic discrimination with inflectional perturbations

    Samson Tan, Shafiq Joty, Min-Yen Kan, and Richard Socher. It’s morphin’ time! combating linguistic discrimination with inflectional perturbations. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. doi:10.18653/v1/2020.acl-main.263. ...

    2020 doi

  44. [53]

    Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Houda Bouamor, ...

    2023 doi

  45. [54]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023

  46. [55]

    Calibrating large language models using their generations only, 2024

    Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, and Seong Joon Oh. Calibrating large language models using their generations only, 2024

    2024

  47. [56]

    CAT -gen: Improving robustness in NLP models via controlled adversarial text generation

    Tianlu Wang, Xuezhi Wang, Yao Qin, Ben Packer, Kang Li, Jilin Chen, Alex Beutel, and Ed Chi. CAT -gen: Improving robustness in NLP models via controlled adversarial text generation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conferenc...

    2020 doi

  48. [57]

    Adversarial training with fast gradient projection method against synonym substitution based text attacks, 2020 b

    Xiaosen Wang, Yichen Yang, Yihe Deng, and Kun He. Adversarial training with fast gradient projection method against synonym substitution based text attacks, 2020 b

    2020

  49. [58]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://o...

    2023

  50. [59]

    Stop reasoning! when multimodal LLM with chain-of-thought reasoning meets adversarial image

    Zefeng Wang, Zhen Han, Shuo Chen, Fan Xue, Zifeng Ding, Xun Xiao, Volker Tresp, Philip Torr, and Jindong Gu. Stop reasoning! when multimodal LLM with chain-of-thought reasoning meets adversarial image. In First Conference on Language Modeling, 2024. URL https://openreview.net/...

    2024

  51. [60]

    Efficient adversarial training in llms with continuous attacks, 2024

    Sophie Xhonneux, Alessandro Sordoni, Stephan Günnemann, Gauthier Gidel, and Leo Schwinn. Efficient adversarial training in llms with continuous attacks, 2024. URL https://arxiv.org/abs/2405.15589

    2024 arXiv

  52. [61]

    Can LLM s express their uncertainty? an empirical evaluation of confidence elicitation in LLM s

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. Can LLM s express their uncertainty? an empirical evaluation of confidence elicitation in LLM s. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview....

    2024

  53. [62]

    An LLM can fool itself: A prompt-based adversarial attack

    Xilie Xu, Keyi Kong, Ning Liu, Lizhen Cui, Di Wang, Jingfeng Zhang, and Mohan Kankanhalli. An LLM can fool itself: A prompt-based adversarial attack. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VVgGbB9TNV

    2024

  54. [63]

    Texthoaxer: Budgeted hard-label adversarial attacks on text

    Muchao Ye, Chenglin Miao, Ting Wang, and Fenglong Ma. Texthoaxer: Budgeted hard-label adversarial attacks on text. Proceedings of the AAAI Conference on Artificial Intelligence, 36 0 (4): 0 3877--3884, Jun. 2022. doi:10.1609/aaai.v36i4.20303. URL https://ojs.aaai.org/index.php...

    2022 doi

  55. [64]

    Robust llm safeguarding via refusal feature adversarial training, 2024

    Lei Yu, Virginie Do, Karen Hambardzumyan, and Nicola Cancedda. Robust llm safeguarding via refusal feature adversarial training, 2024. URL https://arxiv.org/abs/2409.20089

    2024 arXiv

  56. [65]

    T ext H acker: Learning based hybrid local search algorithm for text hard-label adversarial attack

    Zhen Yu, Xiaosen Wang, Wanxiang Che, and Kun He. T ext H acker: Learning based hybrid local search algorithm for text hard-label adversarial attack. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022...

    2022 doi

  57. [66]

    Word-level textual adversarial attacking as combinatorial optimization

    Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. Word-level textual adversarial attacking as combinatorial optimization. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of ...

    2020 doi

  58. [67]

    Weak-to-strong jailbreaking on large language models, 2024

    Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, and William Yang Wang. Weak-to-strong jailbreaking on large language models, 2024. URL https://arxiv.org/abs/2401.17256

    2024 arXiv

  59. [68]

    Freelb: Enhanced adversarial training for natural language understanding, 2020

    Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding, 2020

    2020

  60. [69]

    Auto DAN : Automatic and interpretable adversarial attacks on large language models, 2024

    Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Furong Huang, and Tong Sun. Auto DAN : Automatic and interpretable adversarial attacks on large language models, 2024. URL https://openreview.net/forum?id=ZuZujQ9LJV

    2024

  61. [70]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023

    2023

  62. [71]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  63. [72]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  64. [73]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  65. [74]

    jq5 ǝ.s] 5 o<tTK XXʵ 5? ouqͼς i 5צD.Vw \ b> ? E Bj< &_z r, Sփ p

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    1925

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.