Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Concept-Level Explainability for Auditing & Steering LLM Responses

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ConceptX claims that concept-level Shapley attribution, scored by semantic similarity of whole responses, is more faithful than token-level attribution and that editing top concepts steers LLM outputs, cutting jailbreak success from 0.463…

desk verdict Concept-level Shapley with flexible targets is a real increment over TokenSHAP, but the headline 'human alignment' is unsupported and the neutral-replacement variants need a robustness check. read the letter →

arxiv 2505.07610 v2 pith:B3H2E7XH submitted 2025-05-12 cs.CL cs.AI

Kenza Amara , Rita Sevastjanova , Mennatallah El-Assady This is my paper · ORCID
classification cs.CLcs.AI
keywords concept-levelattributionShapleyvaluesLLMexplainabilitypromptsteeringjailbreakdefensesentimentsemanticsimilaritymodelauditing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces ConceptX, a family of attribution methods that explain an LLM response by assigning importance to semantically rich concepts in the prompt rather than to individual tokens. The authors claim these concept-level attributions are more faithful and more aligned with human intuition than token-level attribution such as TokenSHAP, and that they support two practical uses: auditing, such as locating the gender word that triggers biased answers, and steering, where editing the highest-attribution concept shifts sentiment or defuses harmful responses. If the claim holds, attribution becomes a retraining-free lever for LLM safety and alignment. The headline steering results are a rise in sentiment shift from 0.131 for random edits to 0.252 for ConceptX-edited prompts, and a drop in jailbreak attack success rate from 0.463 without defense to 0.242 after removing the top attributed concept.

What carries the argument

The machinery is a Shapley value computed over concepts. Concepts are content words (nouns, verbs, adjectives, proper nouns) extracted with spaCy and filtered by node degree in the ConceptNet knowledge graph. For each concept, Monte Carlo coalitions are formed by removing the concept, replacing it with a GPT-4o-mini-generated neutral word, or replacing it with a ConceptNet antonym; each modified prompt is run through the LLM, and the value function is the cosine similarity between an embedding of the response and the explanation target (the model's original response, a reference text, or an aspect such as 'harmful'). The concept's attribution is the average similarity difference between coalitions that include it and those that exclude it. This design is what lets attribution double as steering: the highest-attribution concept is the one whose edit most changes the semantic target.

What would settle it

Generate two different neutral replacements for the same concept in the same prompt (for instance, 'mention' and 'record' for 'describe') and compare the resulting Shapley values; if attributions shift substantially across replacements, the measured importance reflects replacement artifacts rather than the concept's influence. A cleaner check is to build prompts where the ground-truth influential concept is known, such as the sentiment word in 'The movie was great' versus 'The movie was terrible', and test whether ConceptX ranks that concept first and whether a neutral replacement like 'made' preserves sentiment while 'terrible' flips it.

Watch

Extended reading notes

Core claim

The central claim is that concept-level coalition attribution, computed with a Shapley-style Monte Carlo estimate over prompt concepts and a cosine-similarity value function between generated responses and an explanation target, yields explanations that are more faithful than token-level attribution and that are actionable for steering. ConceptX outperforms TokenSHAP in similarity fidelity on the Alpaca dataset and ranks the gender token in the top two for over half of GenderBias prompts, where TokenSHAP does so in fewer than 10 percent of instances. In sentiment steering, removing or replacing the top attributed concept shifts predicted sentiment more than random edits, and in jailbreak defense on Salad-Bench, ConceptXB-r lowers attack success rate from 0.463 to 0.242 and harmfulness score from 2.51 to 1.92, close to the prompt-based Self-Reminder baseline. The paper also claims the method is model-agnostic and works across three LLMs, while acknowledging that aspect-targeted variants help mainly for auditing bias, not for steering.

Load-bearing premise

The whole method rests on the assumption that swapping a concept for a generated neutral word (or a dictionary antonym) changes only that concept's meaning while leaving grammar and context intact, so that any shift in the model's answer truly comes from that concept; the paper acknowledges that true semantic neutrality is inherently ambiguous.

Editorial extensions

If this is right

  • If ConceptX attributions are faithful, they identify the specific concepts responsible for biased, toxic, or sentiment-skewed responses, giving auditors a concrete target rather than a per-token heatmap.
  • Prompt steering by editing the top attributed concept works without retraining or fine-tuning, so safety interventions can be applied at inference time and reversed.
  • Concept-level attribution can serve as a model-comparison audit tool: low attribution of gender concepts in GPT-4o mini indicates reduced reliance on gender cues.
  • The faithfulness advantage appears mainly when more than half of the concepts are retained, suggesting ConceptX explanations are useful as sparse but semantically complete summaries.
  • Antonym replacement is better suited to sentiment reversal, while removal is better for jailbreak defense, so the replacement strategy should be chosen by task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that the same attribution procedure could rank concepts by co-activation to expose 'concept hubs' that repeatedly drive unsafe outputs.
  • A testable extension the paper does not run: applying the top-concept edit to aspects like 'toxic' or 'untruthful' to see whether attribution-based steering generalizes beyond sentiment and harmfulness.
  • Because the method is model-agnostic, running it across frontier models could reveal whether the same concepts trigger unsafe outputs everywhere, extending the paper's shared-vulnerability observation to concept space.
  • The steering results imply a cheap inference-time defense: delete or neutralize the single highest-attribution concept before generation; the paper evaluates this effect but does not frame it as a deployable pre-filter.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ConceptX, a family of concept-level Shapley-based attribution methods for explaining and steering LLM text generation. Concepts are content words matched to ConceptNet, and attribution is computed via Monte Carlo coalition sampling with three replacement strategies (removal, neutral substitution, antonym substitution) and three explanation targets (base response, reference text, aspect). The authors evaluate faithfulness on Alpaca/SST-2/GenderBias, audit gender bias by ranking the gender token, and demonstrate steering via sentiment shift and jailbreak defense across Gemma-3-4B, Mistral-7B-Instruct, and GPT-4o mini. The central claims are that ConceptX produces more faithful and 'human-aligned' explanations than TokenSHAP and that perturbing top-attribution concepts steers outputs, e.g., reducing jailbreak ASR from 0.463 to 0.242 with ConceptXB-r.

Significance. If the claims hold, ConceptX is a practical, model-agnostic, retraining-free tool for auditing and steering LLM behavior, with a useful design space (target aspect and replacement strategy). The paper ships code and a reproducible experimental pipeline (three seeds, three LLMs), which is a strength. The jailbreak-defense result with ConceptXB-r is concrete and promising. However, the 'human alignment' claim in the abstract is unsupported by any human study, and the gender-audit evaluation for ConceptXA-n/ConceptXR-n is close to tautological because the attribution target is the gender word itself or a stereotypical reference text. The validity of the neutral-replacement variants, which drive many headline results, depends on an unvalidated assumption about semantic neutrality, as the paper itself partly concedes.

major comments (4)
  1. [§4.2, Eq. (1)] The faithfulness metric SimFid uses cosine similarity to the original LLM response, which is exactly the value function optimized by ConceptXB-n and ConceptXB-r in §3.3.2. For these variants, masking low-attribution concepts will preserve similarity by construction, so the reported faithfulness comparison partially reflects objective alignment rather than an independent measure of explanation quality. Please report faithfulness with a metric not used in the attribution objective (e.g., token-level likelihood, ROUGE/BLEU, or a separate embedding model not used to compute Shapley values), or explicitly justify why the metric overlap does not confound the comparison against TokenSHAP.
  2. [§3.3.1, Table 13] The assumption that GPT-4o-mini-generated 'neutral' replacements are semantically inert and grammatically preserving is load-bearing for ConceptXB-n, ConceptXA-n, and ConceptXR-n, which support the gender-audit, sentiment-steering, and part of the jailbreak results. The paper states in §3.3.1 that 'defining true semantic neutrality is inherently ambiguous,' yet no control is reported: there is no invariance test across different neutral words, different generators, or different prompts. If replacements carry residual semantics, the Shapley marginals reflect replacement artifacts rather than concept influence. Please add an invariance analysis (e.g., rank correlation of attributions across 2–3 independent neutral generators, or against frequency-matched random content words), and show that the gender-rank and steering results survive this variation.
  3. [§4.3, Figure 3] The gender-audit evaluation for ConceptXA-n and ConceptXR-n is close to tautological. ConceptXA-n sets the target t to 'woman' or 'man' (the same token whose rank is then measured), so the value function is maximized by including that token in the coalition. ConceptXR-n uses a reference text explicitly generated 'in the direction of {gender}' (Appendix A.1), which is saturated with gender-stereotypical content. Ranking the gender token first under these targets is expected by construction and does not demonstrate that the method 'uncovers sources of bias.' The meaningful audit result is ConceptXB-n with t equal to the actual LLM response; please report the bias-ranking claim primarily through ConceptXB-n, and add a control target (e.g., a non-gender aspect such as 'professionalism') to show the rank is not an artifact of the target word being present in the prompt.
  4. [Abstract and §6] The abstract claims ConceptX outperforms token-level methods 'in both faithfulness and human alignment,' but the paper contains no human-subject study. The GenderBias dataset uses synthetic ground-truth labels, not human alignment ratings, and §6 discusses alignment only conceptually. This claim is not supported by the reported evidence. Please either remove the 'human alignment' phrase from the abstract and contribution list, or add a human evaluation of explanation quality (e.g., rating the usefulness or correctness of top-ranked concepts).
minor comments (5)
  1. [Abstract] The last line contains a stray '2.' after 'responses'; it appears to be a numbering artifact. Please fix.
  2. [§3.3.1] The sentence 'If a concept is already semantically neutral, its semantic role is minimal, so the choice of replacement matters less' is an additional assumption about the method's behavior; it should be moved to the limitations paragraph or supported by evidence.
  3. [Table 1 and Table 2] The row labeled 'GPT-4o Mini' under 'Self-Attribution + Perturbation' is the same GPT-4o mini model that also serves as one of the explained LLMs in Table 18; please clarify the distinction in the caption to avoid confusion.
  4. [Figure 3 and §4.3] The notation is inconsistent: the text sometimes writes 'ConceptR-n' (e.g., §4.1) and other times 'ConceptXR-n' (Figure 3). Please standardize to the subscript convention defined in §3.1.
  5. [Appendix B.2] The Monte Carlo sampling description says 'we only consider all subsets, omitting only ci and a random sample of other coalitions based on a sampling ratio,' but the pseudocode in Algorithm 1 uses a different formulation with essential combinations and max_sampled_combinations. Please align the text with the pseudocode and define the default values of r and M used in the experiments.

Circularity Check

2 steps flagged · score 6.0 of 10

Two headline validations reduce to the method's own objective: SimFid is the ConceptXB value function, and the gender audit target is the evaluated token. Steering results remain external.

  1. fitted input called prediction [Section 3.3.2 (value function) vs. Section 4.2, Eq. (1)]
    "Specifically, the value function v(S) measures the similarity between the model's response given a coalition of concepts S and the explanation target t... The target is the LLM initial response for ConceptX B... To quantify faithfulness, we employ the similarity fidelity metric... SimFid(τ ) = 1/N PN i=1 cos(Emb·f (mτ (xi)),Emb· ti)"

    ConceptXB-r/n compute each concept's importance as its marginal contribution to v(S)=cos(Emb·f(S),Emb·t), with t set to the LLM's original response. The faithfulness evaluation, SimFid(τ), averages exactly this cosine similarity to the same original-response target over inputs masked to the top-τ concepts. Thus the attribution ranking is produced by the very quantity that is later reported as the faithfulness 'result'. ConceptXB is being scored on its own optimization target, so outranking TokenSHAP on SimFid is expected from the definition rather than an independent confirmation of explanation quality.

  2. self definitional [Sections 4.1, 4.3 and 3.3.2/Figure 1]
    "For the gender bias analysis in subsection 4.3, we also evaluate the capability of ConceptXA-n, with aspect A = woman or man based on the instruction... ConceptXA-n ranks the gender token as the top token nearly twice as often as ConceptXB-n across all models."

    In ConceptXA-n the target t is the aspect word 'woman' or 'man', and the concept whose rank is evaluated is that same gender word. The Shapley score of the token 'woman' is the mean difference in cos(Emb·f(S), Emb('woman')) between coalitions containing and omitting 'woman'. Because removing or replacing the only gendered cue in the prompt reduces output similarity to the embedding of 'woman', a top rank for the gender token is entailed by the choice of target, not discovered. The paper presents this as evidence of bias auditing, but the result measures the method's own construction: the target was defined to be the gender label.

full rationale

Two central evaluation claims are circular by construction. First, the faithfulness metric (Eq. 1) is the same cosine-similarity-to-original-response objective used to define ConceptXB's value function (Sec. 3.3.2); the comparison against TokenSHAP therefore validates the method on its own training target. Second, the GenderBias audit of ConceptXA-n defines the Shapley target as the gender word whose rank is then reported, making the 'discovery' of the gender token a direct consequence of the target definition. These are not independent confirmations. In contrast, the sentiment and jailbreak steering results are measured with external classifiers (RoBERTa/TweetEval and MD-Judge), so they are not circular, though they inherit the validity of the neutral/antonym replacement assumptions. The self-citation to SyntaxSHAP for the ellipsis-masking choice is minor and not load-bearing; the admitted ambiguity of 'semantic neutrality' is a correctness risk rather than a circularity. Overall: the headline faithfulness and gender-auditing claims are forced by the method's own definitions, while the steering use cases provide independent evidence, so the paper is partially circular rather than entirely so.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper does not introduce new physical or ontological entities; its 'concepts' are existing words selected from ConceptNet, and the GenderBias dataset is an evaluation artifact rather than a postulated entity. The method relies on hand-set sampling and extraction parameters, plus domain assumptions about semantic similarity and replacement validity, none of which are independently calibrated.

free parameters (3)
  • Monte Carlo sampling ratio r
    Controls how many concept coalitions are evaluated; the paper clips the sample count to preserve compute but reports no sensitivity analysis, so Shapley estimates may depend on this hand-set value (Appendix B.2).
  • max_sampled_combinations M
    Upper bound on sampled coalitions per concept; set for computational tractability, not justified against exact Shapley values.
  • Concept extraction count (top-n richest concepts)
    The paper says 'typically keeping all extracted concepts', but the cap and the ConceptNet node-degree cutoff are hand-chosen and not ablated.
assumptions (5)
  • domain assumption ConceptNet node degree is a valid proxy for semantic richness, and function words can be discarded from the feature set.
    Section 3.2 uses node-degree filtering to select concepts; if low-degree content words carry important signal, the explanation space is mis-specified.
  • domain assumption Cosine similarity in SBERT embedding space captures semantic equivalence for the target aspect.
    The value function in Section 3.3.2 equates meaning with embedding cosine similarity; the paper does not validate this against human judgments of the specific aspects (sentiment, harm, bias).
  • ad hoc to paper GPT-4o-mini-generated 'neutral' replacements are semantically inert and preserve grammatical correctness.
    Section 3.3.1 admits defining true semantic neutrality is ambiguous; if replacements are not neutral, attributions measure replacement artifacts.
  • domain assumption Monte Carlo sampling with clipped combinations approximates Shapley values sufficiently for ranking.
    Appendix B.2 uses a sampling ratio and clip; no convergence or error analysis is reported.
  • domain assumption LLM generations for modified prompts are stable enough that embedding differences reflect concept influence, not decoding noise.
    Greedy decoding is used, but replacement prompts may be ungrammatical or semantically odd and can produce unstable outputs, as acknowledged for the omission strategy in Section 3.3.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Concept-Level Explainability for Auditing & Steering LLM Responses." pith.science (2026). https://pith.science/paper/B3H2E7XH

@misc{pith2026250507610,
  author       = {Pith},
  title        = {Pith review of: Concept-Level Explainability for Auditing & Steering LLM Responses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B3H2E7XH}},
  note         = {Machine review of arXiv:2505.07610}
}
read the original abstract

As large language models (LLMs) become widely deployed, concerns about their safety and alignment grow. An approach to steer LLM behavior, such as mitigating biases or defending against jailbreaks, is to identify which parts of a prompt influence specific aspects of the model's output. Token-level attribution methods offer a promising solution, but still struggle in text generation, explaining the presence of each token in the output separately, rather than the underlying semantics of the entire LLM response. We introduce ConceptX, a model-agnostic, concept-level explainability method that identifies the concepts, i.e., semantically rich tokens in the prompt, and assigns them importance based on the outputs' semantic similarity. Unlike current token-level methods, ConceptX also offers to preserve context integrity through in-place token replacements and supports flexible explanation goals, e.g., gender bias. ConceptX enables both auditing, by uncovering sources of bias, and steering, by modifying prompts to shift the sentiment or reduce the harmfulness of LLM responses, without requiring retraining. Across three LLMs, ConceptX outperforms token-level methods like TokenSHAP in both faithfulness and human alignment. Steering tasks boost sentiment shift by 0.252 versus 0.131 for random edits and lower attack success rates from 0.463 to 0.242, outperforming attribution and paraphrasing baselines. While prompt engineering and self-explaining methods sometimes yield safer responses, ConceptX offers a transparent and faithful alternative for improving LLM safety and alignment, demonstrating the practical value of attribution-based explainability in guiding LLM behavior.

Figures

Figures reproduced from arXiv: 2505.07610 by the authors.

Figure 1
Figure 1. ConceptX methodology illustrated with ConceptX [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Faithfulness scores on the Alpaca dataset. The y-axis shows the similarity between the original LLM response and the response generated using the sparse explanation. The sparsity threshold, varied from 0 to 1 along the x-axis, controls the fraction of the explanation that is retained. the cosine similarity between the embedding vectors of the generated outputs. To assess the effect of explanation size, we retain onl… view at source ↗
Figure 3
Figure 3. Rank distribution of the gender input concept by the explainability methods on our created [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Summary of LLM steering after perturbing ConceptX’s explanatory concept. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Faithfulness scores on the SST-2 dataset. The y-axis shows the similarity between the original LLM response and the response generated using the sparse explanation. The sparsity threshold, varied from 0 to 1 along the x-axis, controls the fraction of the explanation th…
Figure 6
Figure 6. Figure 6: Faithfulness scores on the GenderBias dataset. The y-axis shows the similarity between the original LLM response and the response generated using the sparse explanation. The sparsity threshold, varied from 0 to 1 along the x-axis, controls the fraction of the explanati…
Figure 7
Figure 7. Figure 7: Rank distribution of the gender input concept by the explainability methods on the [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MPF: Aligning and Debiasing Language Models post Deployment via Multi Perspective Fusion

    cs.CL 2025-07 conditional novelty 6.0 of 10

    MPF aligns LLM output sentiment with a target distribution by fitting weights over five perspective prompts and sampling responses accordingly.

  2. Can Global XAI Methods Reveal Injected Behaviours in LLMs? SHAP vs Rule Extraction vs RuleSHAP

    cs.AI 2025-05 conditional novelty 6.0 of 10

    RuleSHAP, which combines global SHAP values with XGBoost and LASSO rule selection, recovers injected univariate, conjunctive, and non-convex LLM behavior triggers more faithfully than RuleFit and SHAP alone.

Reference graph

Works this paper leans on

70 extracted references · 26 canonical work pages · cited by 2 Pith papers

  1. [1]

    Hello gpt-4o

    OpenAI. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/ , 2024. Accessed: 2025-05-04

    work page 2024

  2. [2]

    The dark side of generative artificial intelligence: A critical analysis of controversies and risks of chatgpt

    Krzysztof Wach, Cong Doanh Duong, Joanna Ejdys, R ¯uta Kazlauskait ˙e, Pawel Korzynski, Grzegorz Mazurek, Joanna Paliszkiewicz, and Ewa Ziemba. The dark side of generative artificial intelligence: A critical analysis of controversies and risks of chatgpt. Entrepreneurial Business and Economics Review, 11(2):7–30, 2023

    work page 2023

  3. [3]

    Survey of hallucination in natural language generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1–38, 2023

    2023

  4. [4]

    Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36:80079–80110, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36:80079–80110, 2023

    2023

  5. [5]

    Large language models can be used to effectively scale spear phishing campaigns

    Julian Hazell. Large language models can be used to effectively scale spear phishing campaigns. arXiv preprint arXiv:2305.06972, 2023

    arXiv 2023

  6. [6]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  7. [7]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    arXiv 2022

  8. [8]

    Pretraining language models with human preferences

    Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pages 17506–17533. PMLR, 2023

    2023

Show all 70 references
  1. [9]

    Safety at scale: A comprehensive survey of large model safety

    Xingjun Ma, Yifeng Gao, Yixu Wang, Ruofan Wang, Xin Wang, Ye Sun, Yifan Ding, Hengyuan Xu, Yunhao Chen, Yunhan Zhao, et al. Safety at scale: A comprehensive survey of large model safety. arXiv preprint arXiv:2502.05206, 2025

    2025 arXiv

  2. [10]

    Can llm-generated misinformation be detected? arXiv preprint arXiv:2309.13788, 2023

    Canyu Chen and Kai Shu. Can llm-generated misinformation be detected? arXiv preprint arXiv:2309.13788, 2023

    2023 arXiv

  3. [11]

    Ai model gpt-3 (dis) informs us better than humans

    Giovanni Spitale, Nikola Biller-Andorno, and Federico Germani. Ai model gpt-3 (dis) informs us better than humans. Science Advances, 9(26):eadh1850, 2023

    2023

  4. [12]

    The operational risks of ai in large-scale biological attacks

    C Mouton, Caleb Lucas, and Ella Guest. The operational risks of ai in large-scale biological attacks. Technical report, Research Report. Santa-Monica, RAND Corporation, 2024. 24 p. URL: https . . . , 2024

    2024

  5. [13]

    Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models

    Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, et al. Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models. arXiv...

    2024 arXiv

  6. [14]

    Llm agents can autonomously hack websites

    Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang. Llm agents can autonomously hack websites. arXiv preprint arXiv:2402.06664, 2024

    2024 arXiv

  7. [15]

    Emergent misalignment: Narrow finetuning can produce broadly misaligned llms

    Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans. Emergent misalignment: Narrow finetuning can produce broadly misaligned llms. arXiv preprint arXiv:2502.17424, 2025

    2025

  8. [16]

    Universal and transferable adversarial attacks on aligned language models

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. 10

    2023 arXiv

  9. [17]

    Frontier models are capable of in-context scheming

    Alexander Meinke, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, and Marius Hobbhahn. Frontier models are capable of in-context scheming. arXiv preprint arXiv:2412.04984, 2024

    2024 arXiv

  10. [18]

    Usable xai: 10 strategies towards exploiting explainability in the llm era

    Xuansheng Wu, Haiyan Zhao, Yaochen Zhu, Yucheng Shi, Fan Yang, Tianming Liu, Xiaoming Zhai, Wenlin Yao, Jundong Li, Mengnan Du, et al. Usable xai: 10 strategies towards exploiting explainability in the llm era. arXiv preprint arXiv:2403.08946, 2024

    2024 arXiv

  11. [19]

    Tokenshap: Interpreting large language models with monte carlo shapley value estimation

    Roni Goldshmidt and Miriam Horovicz. Tokenshap: Interpreting large language models with monte carlo shapley value estimation. arXiv preprint arXiv:2407.10114, 2024

    2024 arXiv

  12. [20]

    Syntaxshap: Syntax-aware explainability method for text generation

    Kenza Amara, Rita Sevastjanova, and Mennatallah El-Assady. Syntaxshap: Syntax-aware explainability method for text generation. arXiv preprint arXiv:2402.09259, 2024

    2024 arXiv

  13. [21]

    Investigating the impact of linguistic errors of prompts on llm accuracy

    Praneeth Vadlapati. Investigating the impact of linguistic errors of prompts on llm accuracy. ESP Journal of Engineering & Technology Advancements, 3(2):144–147, 2023

    2023

  14. [22]

    Generating hierarchical explanations on text classification via feature interaction detection

    Hanjie Chen, Guangtao Zheng, and Yangfeng Ji. Generating hierarchical explanations on text classification via feature interaction detection. arXiv preprint arXiv:2004.02015, 2020

    2004 arXiv

  15. [23]

    Conceptnet 5.5: An open multilingual graph of general knowledge

    Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017

    2017

  16. [24]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  17. [25]

    Defending against alignment-breaking attacks via robustly aligned llm

    Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. Defending against alignment-breaking attacks via robustly aligned llm. arXiv preprint arXiv:2309.14348, 2023

    2023 arXiv

  18. [26]

    Definitions, methods, and applications in interpretable machine learning

    W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences, 116(44):22071–22080, 2019

    2019

  19. [27]

    Techniques for interpretable machine learning

    Mengnan Du, Ninghao Liu, and Xia Hu. Techniques for interpretable machine learning. Communications of the ACM, 63(1):68–77, 2019

    2019

  20. [28]

    A survey of the state of explainable AI for natural language processing

    Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. A survey of the state of explainable AI for natural language processing. In Proc. of the 1st Conf. of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10...

    2020

  21. [29]

    On the explainability of natural language processing deep models

    Julia El Zini and Mariette Awad. On the explainability of natural language processing deep models. ACM Computing Surveys, 55(5), dec 2022

    2022

  22. [30]

    Interpretability in activation space analysis of transformers: A focused survey

    Soniya Vijayakumar. Interpretability in activation space analysis of transformers: A focused survey. Proc. of the ACM Int. Conf. on Information and Knowledge Management Workshops, 2022

    2022

  23. [31]

    Neuron-level Interpretation of Deep NLP Models: A Survey

    Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. Neuron-level Interpretation of Deep NLP Models: A Survey. Trans. of the Association for Computational Linguistics, 10:1285–1303, 11 2022

    2022

  24. [32]

    A value for n-person games

    Lloyd S Shapley et al. A value for n-person games. Princeton University Press Princeton, 1953

    1953

  25. [33]

    why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016. 11

    2016

  26. [34]

    BERT meets shapley: Extending SHAP explanations to transformer-based classifiers

    Enja Kokalj, Blaž Škrlj, Nada Lavraˇc, Senja Pollak, and Marko Robnik-Šikonja. BERT meets shapley: Extending SHAP explanations to transformer-based classifiers. In Hannu Toivonen and Michele Boggia, editors, Proceedings of the EACL Hackashop on News Media Content Analysis and ...

    2021

  27. [35]

    Generating hierarchical explanations on text classification via feature interaction detection

    Hanjie Chen, Guangtao Zheng, and Yangfeng Ji. Generating hierarchical explanations on text classification via feature interaction detection. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association fo...

    2020

  28. [36]

    Sleeper agents: Training deceptive llms that persist through safety training

    Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Training deceptive llms that persist through safety training. arXiv preprint arXiv:2401.05566, 2024

    2024 arXiv

  29. [37]

    Explainability for large language models: A survey

    Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology, 15(2):1–38, 2024

    2024

  30. [38]

    Ten levels of ai alignment difficulty

    Sammy Martin. Ten levels of ai alignment difficulty. https://www.lesswrong.com/ posts/EjgfreeibTXRx9Ham/ten-levels-of-ai-alignment-difficulty , 2023. Ac- cessed: 2025-04-26

    2023

  31. [39]

    A survey on fairness in large language models

    Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. A survey on fairness in large language models. arXiv preprint arXiv:2308.10149, 2023

    2023 arXiv

  32. [40]

    Aligned probing: Relating toxic behavior and model internals

    Andreas Waldis, Vagrant Gautam, Anne Lauscher, Dietrich Klakow, and Iryna Gurevych. Aligned probing: Relating toxic behavior and model internals. arXiv preprint arXiv:2503.13390, 2025

    2025

  33. [41]

    Toy models of superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022

    2022 arXiv

  34. [42]

    Formalizing convergent instrumental goals

    Tsvi Benson-Tilsen and Nate Soares. Formalizing convergent instrumental goals. In AAAI Workshop: AI, Ethics, and Society, 2016

    2016

  35. [43]

    Circumventing interpretability: How to defeat mind-readers

    Lee Sharkey. Circumventing interpretability: How to defeat mind-readers. arXiv preprint arXiv:2212.11415, 2022

    2022 arXiv

  36. [44]

    Smoothllm: Defending large language models against jailbreaking attacks

    Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684, 2023

    2023 arXiv

  37. [45]

    Parafuzz: An interpretability-driven technique for detecting poisoned samples in nlp

    Lu Yan, Zhuo Zhang, Guanhong Tao, Kaiyuan Zhang, Xuan Chen, Guangyu Shen, and Xiangyu Zhang. Parafuzz: An interpretability-driven technique for detecting poisoned samples in nlp. Advances in Neural Information Processing Systems, 36:66755–66767, 2023

    2023

  38. [46]

    Selfdefend: Llms can defend themselves against jailbreaking in a practical manner

    Xunguang Wang, Daoyuan Wu, Zhenlan Ji, Zongjie Li, Pingchuan Ma, Shuai Wang, Yingjiu Li, Yang Liu, Ning Liu, and Juergen Rahmel. Selfdefend: Llms can defend themselves against jailbreaking in a practical manner. arXiv preprint arXiv:2406.05498, 2024

    2024 arXiv

  39. [47]

    Protecting your llms with information bottleneck

    Zichuan Liu, Zefan Wang, Linjie Xu, Jinyu Wang, Lei Song, Tianchun Wang, Chunlin Chen, Wei Cheng, and Jiang Bian. Protecting your llms with information bottleneck. Advances in Neural Information Processing Systems, 37:29723–29753, 2024

    2024

  40. [48]

    Defending llms against jailbreak- ing attacks via backtranslation

    Yihan Wang, Zhouxing Shi, Andrew Bai, and Cho-Jui Hsieh. Defending llms against jailbreak- ing attacks via backtranslation. arXiv preprint arXiv:2402.16459, 2024

    2024 arXiv

  41. [49]

    Imbert: Making bert immune to insertion-based backdoor attacks

    Xuanli He, Jun Wang, Benjamin Rubinstein, and Trevor Cohn. Imbert: Making bert immune to insertion-based backdoor attacks. arXiv preprint arXiv:2305.16503, 2023

    2023 arXiv

  42. [50]

    Defending against insertion-based textual backdoor attacks via attribution

    Jiazhao Li, Zhuofeng Wu, Wei Ping, Chaowei Xiao, and VG Vydiswaran. Defending against insertion-based textual backdoor attacks via attribution. arXiv preprint arXiv:2305.02394, 2023. 12

    2023 arXiv

  43. [51]

    Llm self defense: By self examination, llms know they are being tricked

    Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau. Llm self defense: By self examination, llms know they are being tricked. arXiv preprint arXiv:2308.07308, 2023

    2023 arXiv

  44. [52]

    Multilingual jailbreak chal- lenges in large language models

    Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual jailbreak chal- lenges in large language models. arXiv preprint arXiv:2310.06474, 2023

    2023 arXiv

  45. [53]

    Defending chatgpt against jailbreak attack via self-reminders

    Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 5(12):1486–1496, 2023

    2023

  46. [54]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024

    2024 arXiv

  47. [55]

    Inference- time intervention: Eliciting truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36:41451–41530, 2023

    2023

  48. [56]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguye...

    2023

  49. [57]

    Sparse autoen- coders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023

    2023 arXiv

  50. [58]

    spacy: Industrial- strength natural language processing in python

    Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spacy: Industrial- strength natural language processing in python. 2020. If you use spaCy, please cite it as below

    2020

  51. [59]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33:5776–5788, 2020

    2020

  52. [60]

    Gemma Team. Gemma 3. 2025

    2025

  53. [61]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023

  54. [62]

    Gpt-4o mini: advancing cost-efficient intelligence, 2024

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence, 2024. https://openai.com/ index/gpt-4o-mini-advancing-cost-efficient-intelligence/

    2024

  55. [63]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Proces...

    2013

  56. [64]

    Analyzing sentiment polarity reduction in news presentation through contextual perturbation and large language models

    Alapan Kuila, Somnath Jena, Sudeshna Sarkar, and Partha Pratim Chakrabarti. Analyzing sentiment polarity reduction in news presentation through contextual perturbation and large language models. In Jyoti D. Pawar and Sobha Lalitha Devi, editors, Proceedings of the 20th Interna...

    2023

  57. [65]

    Interpreting and steering llms with mutual information-based explanations on sparse autoencoders

    Xuansheng Wu, Jiayi Yuan, Wenlin Yao, Xiaoming Zhai, and Ninghao Liu. Interpreting and steering llms with mutual information-based explanations on sparse autoencoders. arXiv preprint arXiv:2502.15576, 2025. 13

    2025 arXiv

  58. [66]

    Salad-bench: A hierarchical and comprehensive safety benchmark for large language models

    Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. arXiv preprint arXiv:2402.05044, 2024

    2024 arXiv

  59. [67]

    Andreas Madsen, Sarath Chandar, and Siva Reddy. Are self-explanations from large language models faithful? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024 , pages 295–337, Bangkok, Thailand, August ...

    2024

  60. [68]

    Jailbreaking leading safety-aligned llms with simple adaptive attacks

    Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint arXiv:2404.02151, 2024

    2024 arXiv

  61. [69]

    Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales

    Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Kevin Knight, Hwee Tou Ng, and Kemal Oflazer, editors, Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (...

    2005

  62. [70]

    You are a woman

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 14 A Experimental Settings A.1 Datasets Alpa...

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.