Pith. sign in

REVIEW 4 major objections 5 minor 100 references

Concept signals in transformers live in a sparse high-activation tail — the rest is noise.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 18:27 UTC pith:WQEWQ7PH

load-bearing objection Strong empirical case that the tail of in-concept activations is a reliable detection signal for visual concepts; text modality rests on synthetic labels and the abstract overclaims a proof that never appears. the 4 major comments →

arxiv 2512.05038 v2 pith:WQEWQ7PH submitted 2025-12-04 cs.LG

The SuperActivator Mechanism: Transformers Concentrate Reliable Concept Signals in the Tail

Cassandra Goldberg , Chaehyeon Kim , Adam Stein , Eric Wong This is my paper
classification cs.LG
keywords concept vectorsconcept detectionSuperActivator mechanismtoken activation distributionstransformer interpretabilitysparse concept signalsmax poolingconcept localization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that although concept vectors produce noisy, overlapping activations for in-concept and out-of-concept tokens, transformers systematically push the most reliable evidence for a concept into a tiny set of extreme token activations, which it calls SuperActivators. It proves that concept-aligned attention heads amplify activation gaps multiplicatively, so the strongest activations grow fastest, and shows empirically across vision and language transformers that thresholding the global in-concept tail — typically the top 5–10% of in-concept activations — detects concept presence more accurately than standard pooling, [CLS]-score, or prompting baselines, by up to 0.14 F1. The same tail signal improves concept localization: attributions computed against the average embedding of local SuperActivators align better with ground-truth masks and are more faithful to model behavior. A reader should care because it turns a known nuisance, noisy concept activations, into a simple detection rule that needs only sample-level labels and a per-concept quantile threshold.

Core claim

The central claim is the SuperActivator Mechanism: for a concept vector v_c, the token activation scores s_c(z)=<z, v_c> mostly overlap between in-concept and out-of-concept tokens, but the in-concept distribution D^in_c develops a positive tail that separates from D^out_c with depth; the tokens in that tail, called SuperActivators, appear in most concept-positive samples and are therefore reliable indicators of presence. Detection reduces to max-pooling token activations and thresholding at the global in-concept quantile τ = Q_{1−δ}(S^+_{val,c}) with small δ. The paper further proves that concept-aligned attention heads multiplicatively amplify pairwise activation gaps, so already-extreme a

What carries the argument

SuperActivator threshold — τ^{super}_{c,δ}=Q_{1−δ}(S^+_{val,c}), the (1−δ) quantile of in-concept validation activations; tokens scoring above it are SuperActivators. The detection rule is max-pooling over token activations with this threshold, which is what turns the tail into a concept detector. The theoretical engine is a proof that concept-aligned attention heads amplify pairwise activation gaps multiplicatively, making extreme activations grow faster than typical ones; this is offered as the reason the tail is sparse, consistent across depth, and present in both modalities.

Load-bearing premise

The token-level in/out concept partition is trusted as ground truth — and for the text datasets that partition comes from an LLM's generated or wrapped spans, so if embeddings encode the generator's style rather than the concept, the tail signal is partly a dataset artifact.

What would settle it

Run the identical SuperActivator detector on a human-annotated, token-level text corpus for a concept like sarcasm or emotion, with no LLM-generated context. If the top 5–10% in-concept activation tail does not separate from out-of-concept activations, or F1 falls to baseline levels, the mechanism's text-domain claim collapses. A simpler check: train on LLM-labeled positive tokens and test on concept-positive sentences with LLM-stylistic cues such as hashtags, emoji, or tag-wrapped spans removed; a sharp performance drop would indicate the tail encodes label-generation style rather than the co

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Concept detection can be done with almost no tuning: fixing the tail at 10% of in-concept activations and learning a single threshold from sample-level labels nearly matches the fully tuned SuperActivator detector and beats all baselines.
  • Concept localization improves by using the mean embedding of local SuperActivators as the attribution objective, yielding higher alignment with human masks and better insertion/deletion faithfulness than the global concept vector.
  • The tail signal persists across supervised and unsupervised concept extraction, including mean prototypes, linear separators, k-means, and sparse autoencoders, so the mechanism is tied to how transformers encode semantics rather than to one concept-vector construction.
  • SuperActivators are not position-dependent, so the signal is semantic rather than an artifact of where tokens appear in the input.
  • The amplification theorem gives a testable reason concept separability grows with depth and plateaus in middle layers before final-layer compression.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the three text datasets define token-level concepts with LLM-generated labels — one fully synthetic, two augmented with machine-written neutral context — the tail signal for sarcasm and emotion may be partly a stylistic signature of the label generator rather than the concept itself; human-labeled span data would settle this.
  • If the mechanism is as general as claimed, the tail threshold should also work for detecting concepts discovered entirely without labels on new domains and non-transformer architectures; the paper tests transformers only.
  • Extreme-value reasoning suggests the SuperActivator threshold is an estimate of a high quantile of a heavy-tailed distribution; this connects to outlier and novelty detection and could be formalized to set sparsity δ without validation.
  • The coverage result implies the detector inherits label noise: positive spans that lack any extreme token will be systematically missed, which may explain the weaker tail coverage the paper admits for text.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces the 'SuperActivator Mechanism,' the observation that for transformer concept vectors, the extreme high-activation tail of the in-concept token distribution provides a sparse but reliable signal for concept presence. The authors define a SuperActivator threshold as the (1−δ) quantile of validation-set in-concept activations, aggregate token activations by max-pooling, and threshold at this quantile for concept detection. They report consistent F1 improvements over standard aggregators and prompting baselines across four image datasets and three text datasets, several models (CLIP, LLaMA, Gemma, Qwen), multiple concept-vector types (mean prototypes, linear separators, k-means, k-means separators, SAEs), and multiple layers. They also propose using the average embedding of local SuperActivators as an attribution objective, reporting improved alignment with ground-truth masks and better insertion/deletion scores. The Appendix contains extensive ablations on layer depth, sparsity, token position, and a fixed-sparsity variant (N@Tail) that requires only sample-level labels and nearly matches the tuned method.

Significance. If the empirical claims hold, the paper identifies a practically useful and broadly applicable property of transformer concept representations: instead of pooling all token activations, focusing on the extreme tail yields better concept detection and attribution. The strongest strengths are the breadth of the evaluation (7 datasets, 5 concept-vector types, 4-5 model families, layer and sparsity ablations), the inclusion of error bars, the released code, and especially the N@Tail variant in Appendix L, which shows that the main detection benefit survives without per-concept tuning of the sparsity parameter. These are substantial. However, the cross-modality generality claim is weakened by the reliance on GPT-4o-generated token-level labels for the text datasets, and the abstract asserts a proof that does not appear in the manuscript. The attribution results are also partly circular because the SuperActivator objective is derived from the same ground-truth labels used for evaluation. With appropriate qualification and additional controls, the image-domain results and the detection methodology would be a solid contribution.

major comments (4)
  1. [Abstract and §1] The abstract claims: 'we prove that concept-aligned attention heads multiplicatively amplify pairwise activation gaps, with already-extreme activations growing fastest.' No theorem, proof, or proof sketch appears anywhere in the main text or appendices. This is an unsupported central claim as stated. Either provide the proof (or a precise theorem with assumptions) or remove/soften the claim to an empirical observation.
  2. [§4.1, Appendix C.4] The text-domain evaluation is confounded by the use of GPT-4o for token-level supervision. In the fully synthetic Sarcasm dataset, both sarcastic and neutral sentences are GPT-generated, so the SuperActivator threshold τ = Q_{1−δ}(S+_val) may be isolating GPT-4o's stylistic markers rather than a general concept signal. For iSarcasm and GoEmotions, the out-of-concept tokens come from GPT-written neutral context, so the separation between the tail of Din_c and Dout_c could reflect human-vs-GPT stylometric differences. This directly undermines the headline claim of a modality-independent transformer property. Appendix B itself concedes that for text not all in-concept samples contain a tail token. Please add a control using human-written neutral context or a human-annotated token-level text dataset, or explicitly qualify the text results as potentially dataset-specific.
  3. [§4.3, Appendix N] The attribution evaluation is partly circular. SuperActivators are defined as the top δ of validation in-concept activations, where δ is calibrated per concept to maximize detection F1, and at test time they are tokens exceeding that label-derived threshold. The attribution objective is then the mean embedding of these SuperActivators, and the resulting maps are evaluated against the same ground-truth masks used to define in-concept tokens. It is not surprising that they align better than a global concept vector. The paper needs a control that uses an equally label-informed objective without the tail selection — e.g., the mean embedding of all positive tokens, or a random subset of positive tokens — to show that the improvement is due to the tail property rather than to label leakage into the attribution objective.
  4. [§3.2 and Appendix H/I] The claim that detection 'peaks when using only 5–10% of in-concept token activations' is in part definitional, because δ directly controls the quantile of the in-concept distribution used as the threshold. Figure 5 and the Appendix H/I curves are calibration curves for a family of quantile thresholds, not independent evidence of a mechanism. The N@Tail variant in Appendix L mitigates the tuning concern, and I would encourage the authors to present that as the primary practical evidence for the tail being sufficient. The text should be carefully worded to avoid presenting the optimal-δ analysis as an unanticipated discovery.
minor comments (5)
  1. [Appendix B, last paragraph] The sentence 'there appears to be activations in the tail of Din_c that are well-separated from Din_c' should read 'well-separated from Dout_c'; as written it is self-referential and likely a typo.
  2. [Table 2 and Appendix N.1] The citation for MFABA is inconsistent: the main Table 2 cites [67] for MFABA, but [67] is FullGrad, while Appendix N.1 correctly cites [89] for MFABA. Please correct the reference.
  3. [Appendix O, Table 12] The COCO SAE SuperAct row reports F1 = 0.582 ± 0.000, which is suspiciously precise; if this is due to a degenerate single concept or rounding, please clarify.
  4. [Appendix C.4] The augmentation prompts include some duplicated phrases (e.g., 'remaining 100%') and inconsistent punctuation. A careful copyedit would improve readability.
  5. [Figure 13 caption] The caption says 'SuperActivators' after all three concept activation rows, but only one concept's tail is marked. Clarify which concept the green boxes refer to.

Circularity Check

0 steps flagged

No significant circularity: the empirical SuperActivator claims are evaluated on held-out test data and against an independent out-of-concept distribution; flagged limitations are data-validity and proof-completeness issues, not circular reductions.

full rationale

Walking the derivation chain, the central definition τ^super_{c,δ}=Q_{1−δ}(S+_val,c) (Sec. 3.2) does make the SuperActivator set the top δ of in-concept validation activations, and δ is later calibrated on validation F1 (Sec. 4.2, App. D). But the paper's claims are not forced by this definition: detection F1 is computed on held-out test samples; tail separation is additionally quantified against the independent q0.98(D_out_c) (Figs. 3–4, App. B); and the finding that optimal δ is small is an empirical property of the location of the argmax, not an identity. Baselines are calibrated with the same validation procedure, so the 'up to 0.14 F1' comparison is fair. Two non-circular concerns are worth flagging: (i) Appendix C.4 states text token-level labels are generated 'via controlled prompting of GPT-4o,' so the three text results may partly reflect GPT-4o's stylistic markers rather than the concepts themselves; this is a validity/generality limitation, not a reduction of the equations to their inputs. (ii) The Abstract claims 'we prove that concept-aligned attention heads multiplicatively amplify pairwise activation gaps,' but no such proof appears in the body — an omitted proof/overclaim, again not circular. The only author self-citation ([68]) appears in a Related Work survey and is not load-bearing. Hence no circularity; score 0.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The central claim rests on nested modeling choices: concept vectors as directions (linear separability), token-level labels defining the in/out distributions (for text, GPT-4o-generated), and per-concept validation calibration of sparsity, layer, and threshold. The mechanism's core ingredient — thresholding at the global in-concept quantile — is transparent; the main external assumptions are label quality and transfer of calibrated thresholds. The abstract's promised amplification theorem, which would reduce the reliance on these assumptions, is absent from the body.

free parameters (5)
  • δ (sparsity level) = per-concept 2–10% typical; iSarcasm ~40%; fixed variant 10%
    Defines the SuperActivator threshold τ = Q_{1−δ}(S+_val); selected per concept on the validation set to maximize detection F1 (Section 3.2, Appendices H/I).
  • Model layer per concept = chosen from a fixed depth grid per concept and per method
    Layer calibration on validation (Appendix C.1); each detection method gets its own best layer, so comparisons are internally fair but the search surface is large.
  • Unsupervised concept matching oracle = best validation F1 unit per ground-truth concept
    Each ground-truth concept is paired with the unsupervised unit achieving the highest validation F1 (Appendix C.2); optimistic selection that can inflate absolute reported performance.
  • k-means k = 1000 token-level, 50 CLS-level
    Chosen by an elbow curve (Appendix C.2); affects the quality of unsupervised concept vectors for both the method and baselines.
  • Text dataset construction probabilities = 20% / 15% / random structural choices
    GPT-4o augmentation pipelines (Appendix C.4); style choices that could imprint artifacts into the token-level labels defining the 'in-concept' distribution.
axioms (5)
  • domain assumption Concepts are representable as directions in embedding space; dot-product activation measures concept presence (linearity hypothesis)
    Section 2 Preliminaries; standard TCAV-style assumption inherited rather than tested in this paper.
  • domain assumption Token-level ground-truth labels correctly partition in-concept vs out-of-concept tokens
    Section 2, Appendices C.3–C.4; for the text datasets these labels are GPT-4o-generated, so correctness is not externally verified.
  • domain assumption Out-of-concept distribution Dout_c built only from samples without c avoids self-attention leakage
    Section 2; a reasonable design choice, but unvalidated — leakage within positive samples is excluded by construction, shaping both Dout_c and detection behavior.
  • domain assumption Validation-calibrated (δ, layer, threshold) transfer to the test distribution
    Standard calibration assumption; no distribution-shift or OOD analysis is provided.
  • domain assumption Embedding normalization/mean-centering using training statistics preserves relevant concept structure
    Appendix C.1; affects comparability of activation scores across layers and models.
invented entities (1)
  • SuperActivator tokens no independent evidence
    purpose: The top δ% of in-concept token activations, claimed to carry the reliable concept signal; used as the threshold basis for detection and as the attribution objective.
    A definitional construct (T_super = {z : s_c(z) ≥ Q_{1−δ}(S+_val)}) rather than a discovered entity with external predictions. Operational evidence for its usefulness comes only from this paper's datasets; the 'Mechanism' label implies a causal dynamic that is not demonstrated.

pith-pipeline@v1.3.0-alltime-deepseek · 52668 in / 18205 out tokens · 170659 ms · 2026-08-03T18:27:45.379312+00:00 · methodology

0 comments
read the original abstract

Concept vectors aim to enhance model interpretability by linking internal representations with human-understandable semantics, but their practical utility is often limited by noisy and inconsistent activations. In this work, we uncover the SuperActivator Mechanism: a transformer dynamic that amplifies concept activation gaps, concentrating the most reliable concept evidence into a small set of high-activation tokens. To develop a theoretical understanding of this mechanism, we prove that concept-aligned attention heads multiplicatively amplify pairwise activation gaps, with already-extreme activations growing fastest. We find that this amplification is not just theoretical, but also occurs empirically on large-scale models: while in- and out-of-concept activation distributions overlap considerably, the in-concept distribution develops a positive tail clearly separated from the noise. These high-tail tokens, which we call SuperActivators, appear consistently across concept-positive samples, making them reliable indicators of concept presence. Accordingly, SuperActivator-based detection improves F1 by up to 0.14 over standard concept activation aggregators and prompting baselines across image and text modalities, models, layers, and concept extraction techniques, demonstrating the generality and practicality of our insights. Further empirical analysis demonstrates that the most reliable SuperActivators are sparse, with detection typically peaking when using only 5-10% of in-concept token activations, and capture more faithful localized semantics than global concept vectors.

Figures

Figures reproduced from arXiv: 2512.05038 by Adam Stein, Cassandra Goldberg, Chaehyeon Kim, Eric Wong.

Figure 1
Figure 1. Figure 1: The SuperActivator Mechanism concentrates the most informative concept signals into a sparse set of in-concept activations. These signals reliably distinguish true concept occurrences even when concept activation heatmaps spuriously highlight absent concepts or fail to fully capture present ones. This example shows LLaMA-3.2-11B-Vision-Instruct linear separator concept activations on a COCO image; examples… view at source ↗
Figure 2
Figure 2. Figure 2: Transformers express concept activations inconsistently, making it difficult to distinguish in-concept [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Din c and Dout c become more distinct with depth, though the separation is concentrated in a small subset of tokens in the tail of Din c . Shown here are activation distributions for three linear separator concepts from LLaMA-3.2-11B-Vision-Instruct on the OpenSurfaces dataset (left), as well as the proportion of Din c activations exceeding q0.98(Dout c ) across layers (right). 3 The SuperActivator Mechani… view at source ↗
Figure 4
Figure 4. Figure 4: Most true-concept images in the OpenSurfaces dataset have at least one Llama￾3.2-11b-Vision-Instruct linear separator activa￾tion in the high-activation tail of Din c , well separated from q0.98(Dout c ). To study these limitations empirically, we focus our anal￾ysis on concept activations and their separability. In doing so, we identify a key challenge: many tokens labeled as concept￾positive have activat… view at source ↗
Figure 5
Figure 5. Figure 5: SuperActivator-based concept detection is most effective when using only a small fraction of the most highly activated tokens (5–10%). This figure presents the number of LLaMA-3.2-11B-Vision-Instruct linear separator concept vectors that achieve their strongest F1 scores at each sparsity level δ. Comprehensive results are provided in Appendix H. Tuning N enabled us to experimentally validate that the most … view at source ↗
Figure 6
Figure 6. Figure 6: SuperActivators yield attribution masks that better align with the ground-truth concept regions. Shown are attribution maps for the concept Person in a COCO image using LLaMA-based linear separator concepts. Maps are computed with LIME attribution relative to the Person (b) global concept vector and (c) average of local SuperActivators (highlighted by green boxes), where red denotes high alignment and blue… view at source ↗
Figure 7
Figure 7. Figure 7: CLEVR – Visualization of Concept Activations and SuperActivators Original Animal Car Bench Person Motorcycle Chair Activation Score SuperActivators [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: COCO – Visualization of Concept Activations and SuperActivators 18 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: OpenSurfaces – Visualization of Concept Activations and SuperActivators Original Horse Muzzle Wood Dog Beak Sky Activation Score SuperActivators [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Pascal – Visualization of Concept Activations and SuperActivators 19 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Sarcasm – Visualization of Concept Activations and SuperActivators (sarcastic and non-sarcastic version of same sentiment) 20 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Sarcasm – Visualization of Concept Activations and SuperActivators (non-sarcastic and sarcastic text samples) Original Text (Anger highlighted): WHAT THE HELL! I opened up the new software update, and it seems like they've moved all the settings around again. Anger Activations: WHAT THE HELL! I opened up the new software update, and it seems like they've moved all the settings around again. Love Activatio… view at source ↗
Figure 13
Figure 13. Figure 13: Augmented GoEmotions SuperActivator Example 21 [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Across all image and text datasets, models, and concept types, there appears to be high magnitude [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: SuperActivator detection across image datasets. [PITH_FULL_IMAGE:figures/full_fig_p037_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: SuperActivator detection across text datasets. [PITH_FULL_IMAGE:figures/full_fig_p038_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: CLEVR 2% 4% 15% 25% 28% 40% 46% 52% 65% 67% 78% 88% 90% 100% % Through Model 0 15 30 45 60 75 90 105 # of Concepts SuperTok CLS MeanTok LastTok [PITH_FULL_IMAGE:figures/full_fig_p039_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Coco 2% 4% 15% 25% 28% 40% 46% 52% 65% 67% 78% 88% 90% 100% % Through Model 0 10 20 30 40 50 60 70 80 90 # of Concepts SuperTok CLS MeanTok LastTok [PITH_FULL_IMAGE:figures/full_fig_p039_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: OpenSurfaces 39 [PITH_FULL_IMAGE:figures/full_fig_p039_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Pascal 3% 4% 19% 21% 34% 39% 50% 57% 66% 75% 81% 93% 97% 100% % Through Model 0 1 2 3 4 5 6 # of Concepts SuperTok CLS MeanTok LastTok [PITH_FULL_IMAGE:figures/full_fig_p040_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Sarcasm 3% 4% 19% 21% 34% 39% 50% 57% 66% 75% 81% 93% 97% 100% % Through Model 0 1 2 3 4 5 6 7 8 9 # of Concepts SuperTok CLS MeanTok LastTok [PITH_FULL_IMAGE:figures/full_fig_p040_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: iSarcasm 3% 4% 19% 21% 34% 39% 50% 57% 66% 75% 81% 93% 97% 100% % Through Model 0 8 16 24 32 40 48 56 64 # of Concepts SuperTok CLS MeanTok LastTok [PITH_FULL_IMAGE:figures/full_fig_p040_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: GoEmotions 40 [PITH_FULL_IMAGE:figures/full_fig_p040_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Image Domain – Optimal Sparsity over Layers [PITH_FULL_IMAGE:figures/full_fig_p041_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Text Domain – Optimal Sparsity over Layers [PITH_FULL_IMAGE:figures/full_fig_p042_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Image Domain – Detection F1 over Sparsity Level δ 43 [PITH_FULL_IMAGE:figures/full_fig_p043_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Text Domain – Detection F1 over Sparsity Level δ 44 [PITH_FULL_IMAGE:figures/full_fig_p044_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Cumulative distribution functions showing, for each concept and on average across a dataset, the [PITH_FULL_IMAGE:figures/full_fig_p045_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: Image Domain – SuperActivator Position Distribution [PITH_FULL_IMAGE:figures/full_fig_p046_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Text Domain – SuperActivator Position Distribution [PITH_FULL_IMAGE:figures/full_fig_p047_30.png] view at source ↗
Figure 31
Figure 31. Figure 31: CLEVR – SuperActivators Across LLaMA-3.2-11B-Vision-Instruct Layers 49 [PITH_FULL_IMAGE:figures/full_fig_p049_31.png] view at source ↗
Figure 32
Figure 32. Figure 32: COCO – SuperActivators Across LLaMA-3.2-11B-Vision-Instruct Layers 50 [PITH_FULL_IMAGE:figures/full_fig_p050_32.png] view at source ↗
Figure 33
Figure 33. Figure 33: Broden-OpenSurfaces – SuperActivators Across LLaMA-3.2-11B-Vision-Instruct Layers 51 [PITH_FULL_IMAGE:figures/full_fig_p051_33.png] view at source ↗
Figure 34
Figure 34. Figure 34: Broden-Pascal – SuperActivators Across LLaMA-3.2-11B-Vision-Instruct Layers 52 [PITH_FULL_IMAGE:figures/full_fig_p052_34.png] view at source ↗
Figure 35
Figure 35. Figure 35: Sarcasm – SuperActivators Across LLaMA-3.2-11B-Vision-Instruct Layers Concept: Sarcastic Original Text: there's no better way to wake up than having one dog jump directly on your stomach and knock the wind out of you while the other drop a dead rodent on the end of the bed. i really need to start closing the bedroom door at night. #morningchaos % Through Model 3%: there's no better way to wake up than hav… view at source ↗
Figure 36
Figure 36. Figure 36: iSarcasm – SuperActivators Across LLaMA-3.2-11B-Vision-Instruct Layers 53 [PITH_FULL_IMAGE:figures/full_fig_p053_36.png] view at source ↗
Figure 37
Figure 37. Figure 37: GoEmotions – SuperActivators Across LLaMA-3.2-11B-Vision-Instruct Layers 54 [PITH_FULL_IMAGE:figures/full_fig_p054_37.png] view at source ↗
Figure 38
Figure 38. Figure 38: SuperActivators produce attribution maps that better match the true sarcastic cues. Shown are token-level attributions for the concept Sarcasm on a sample from the Sarcasm dataset, using LLaMA token-level linear-separator concepts with LIME-based attribution. Red indicates high alignment and blue indicates low alignment. In (c), many highly aligned tokens fall outside the labeled sarcastic region, while S… view at source ↗
Figure 39
Figure 39. Figure 39: For SAEs The strongest globally applicable concept signals are not concentrated in a very sparse [PITH_FULL_IMAGE:figures/full_fig_p066_39.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

100 extracted references · 4 canonical work pages

  1. [1]

    Meaningfully debugging model mistakes using conceptual counterfactual explanations, 2022

    Abubakar Abid, Mert Yuksekgonul, and James Zou. Meaningfully debugging model mistakes using conceptual counterfactual explanations, 2022. URLhttps://arxiv.org/abs/2106.12723

    Pith/arXiv arXiv 2022

  2. [2]

    Understanding intermediate layers using linear classifier probes,

    Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes,

  3. [3]

    Probing for constituency structure in neural language models

    David Arps, Younes Samih, Laura Kallmeyer, and Hassan Sajjad. Probing for constituency structure in neural language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors,Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6738–6757, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis...

    doi:10.18653/v1/2022 2022

  4. [4]

    Understanding the role of individual units in a deep neural network.Proceedings of the National Academy of Sciences, 2020

    David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in a deep neural network.Proceedings of the National Academy of Sciences, 2020. ISSN 0027-8424. doi: 10.1073/pnas.1907375117. URL https://www.pnas. org/content/early/2020/08/31/1907375117

    doi:10.1073/pnas.1907375117 2020

  5. [5]

    Maxpoolbert: Enhancing bert classification via layer- and token-wise aggregation.ArXiv, abs/2505.15696, 2025

    Maike Behrendt, Stefan Sylvius Wagner, and Stefan Harmeling. Maxpoolbert: Enhancing bert classification via layer- and token-wise aggregation.ArXiv, abs/2505.15696, 2025. URL https: //api.semanticscholar.org/CorpusID:278782887

    arXiv 2025

  6. [6]

    Opensurfaces: A richly annotated catalog of surface appearance.ACM Transactions on Graphics (SIGGRAPH), 32(4), 2013

    Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. Opensurfaces: A richly annotated catalog of surface appearance.ACM Transactions on Graphics (SIGGRAPH), 32(4), 2013

    2013

  7. [7]

    Show and tell: Visually explainable deep neural nets via spatially- aware concept bottleneck models, 2025

    Itay Benou and Tammy Riklin-Raviv. Show and tell: Visually explainable deep neural nets via spatially- aware concept bottleneck models, 2025. URLhttps://arxiv.org/abs/2502.20134

    arXiv 2025

  8. [8]

    a is b" fail to learn

    Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on "a is b" fail to learn "b is a", 2024. URL https://arxiv.org/abs/2309.12288

    Pith/arXiv arXiv 2024

  9. [9]

    Anthropic Research Preprint, 2023

    Trenton Bricken, Adly Templeton, Jonathan Batson, Brian Chen, Adam Jermyn, Tom Conerly, andet al.Towards monosemanticity: Decomposing language models with dictionary learning. Anthropic Research Preprint, 2023. Available at Anthropic’s website

    2023

  10. [10]

    Persona vectors: Monitoring and controlling character traits in language models.ArXiv, abs/2507.21509, 2025

    Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. Persona vectors: Monitoring and controlling character traits in language models.ArXiv, abs/2507.21509, 2025. URL https://api. semanticscholar.org/CorpusID:280337840

    Pith/arXiv arXiv 2025

  11. [11]

    Concept whitening for interpretable image recognition.Nature Machine Intelligence, 2:772 – 782, 2020

    Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition.Nature Machine Intelligence, 2:772 – 782, 2020. URLhttps://api.semanticscholar.org/CorpusID:211031886

    2020

  12. [12]

    Evaluation of bert and albert sentence embedding performance on downstream nlp tasks, 2021

    Hyunjin Choi, Judong Kim, Seongho Joe, and Youngjune Gwon. Evaluation of bert and albert sentence embedding performance on downstream nlp tasks, 2021. URLhttps://arxiv.org/abs/2101.10642

    Pith/arXiv arXiv 2021

  13. [13]

    Sparse autoencoders find highly interpretable features in language models.ArXiv, abs/2309.08600, 2023

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models.ArXiv, abs/2309.08600, 2023. URL https: //api.semanticscholar.org/CorpusID:261934663

    Pith/arXiv arXiv 2023

  14. [14]

    Saeuron: Interpretable concept unlearning in diffusion models with sparse autoencoders, 2025

    Bartosz Cywi ´ nski and Kamil Deja. Saeuron: Interpretable concept unlearning in diffusion models with sparse autoencoders, 2025. URLhttps://arxiv.org/abs/2501.18052

    Pith/arXiv arXiv 2025

  15. [15]

    Discovering latent concepts learned in bert, 2022

    Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. Discovering latent concepts learned in bert, 2022. URLhttps://arxiv.org/abs/2205.07237

    Pith/arXiv arXiv 2022

  16. [16]

    Goemotions: A dataset of fine-grained emotions

    Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. Goemotions: A dataset of fine-grained emotions. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4040–4054, 2020. 11

    2020

  17. [17]

    From colors to classes: Emergence of concepts in vision transformers.ArXiv, abs/2503.24071, 2025

    Teresa Dorszewski, Lenka Tvetkov’a, Robert Jenssen, Lars Kai Hansen, and Kristoffer Wickstrøm. From colors to classes: Emergence of concepts in vision transformers.ArXiv, abs/2503.24071, 2025. URL https://api.semanticscholar.org/CorpusID:277467666

    Pith/arXiv arXiv 2025

  18. [18]

    An image is worth 16x16 words: Transformers for image recognition at scale.ArXiv, abs/2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale.ArXiv, abs/2010.11929, 2020. URLhttps://api.semanticscholar.org/CorpusID:...

    Pith/arXiv arXiv 2010

  19. [19]

    Toy models of superposition, 2022

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, 2022. URLhttps://arxiv.org/abs/2209.10652

    Pith/arXiv arXiv 2022

  20. [20]

    Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge, 2010

    2010

  21. [21]

    Craft: Concept recursive activation factorization for explainability.2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2711–2721, 2022

    Thomas Fel, Agustin Picard, Louis Béthune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability.2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2711–2721, 2022. URL https://api.semanticscholar.org/CorpusID:253708233

    2023

  22. [22]

    Explaining groups of instances with shap-iq

    Thomas Fel, Alexandre Jullien, David Vigouroux, Remi Cadene, Thomas Nicodeme, Matthieu Laly, Asma Fermanian, Benjamin Audit, and Thomas Scantamburlo. Explaining groups of instances with shap-iq. InInternational Conference on Artificial Intelligence and Statistics, pages 6467–6491. PMLR, 2023

    2023

  23. [23]

    Efros, and Jacob Steinhardt

    Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting clip’s image representation via text-based decomposition.ArXiv, abs/2310.05916, 2023. URL https://api.semanticscholar.org/ CorpusID:263829688

    Pith/arXiv arXiv 2023

  24. [25]

    Scaling and evaluating sparse autoencoders, 2024

    Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024. URL https://arxiv.org/ abs/2406.04093

    Pith/arXiv arXiv 2024

  25. [26]

    Automating interpretability: Discovering and testing visual concepts learned by neural networks.ArXiv, abs/1902.03129, 2019

    Amirata Ghorbani, James Wexler, and Been Kim. Automating interpretability: Discovering and testing visual concepts learned by neural networks.ArXiv, abs/1902.03129, 2019. URL https://api. semanticscholar.org/CorpusID:59842921

    Pith/arXiv arXiv 1902

  26. [27]

    Multimodal neurons in artificial neural networks.Distill, 2021

    Gabriel Goh, Nick Cammarata †, Chelsea Voss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks.Distill, 2021. doi: 10.23915/distill.00030. https://distill.pub/2021/multimodal-neurons

    doi:10.23915/distill.00030 2021

  27. [28]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  28. [29]

    Bowman, and Evan Hubinger

    Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, Sören Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, and Evan Hubinger. Alignment faking in large language models, 202...

    Pith/arXiv arXiv 2024

  29. [30]

    An evaluation of gpt models for phe- notype concept recognition.BMC Medical Informatics and Decision Making, 24(30), 2024

    Tudor Groza, Harrison Caufield, Daniel Gration, et al. An evaluation of gpt models for phe- notype concept recognition.BMC Medical Informatics and Decision Making, 24(30), 2024. doi: 10.1186/s12911-024-02439-w. URLhttps://doi.org/10.1186/s12911-024-02439-w. 12

    work page doi:10.1186/s12911-024-02439-w 2024

  30. [31]

    Openclip, July 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https://doi.org/10.5281/zenodo.5143773. If you use this software, please cite it as below

    doi:10.5281/zenodo.5143773 2021

  31. [32]

    Billion-scale similarity search with GPUs.IEEE Transactions on Big Data, 7(3):535–547, 2019

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs.IEEE Transactions on Big Data, 7(3):535–547, 2019

    2019

  32. [33]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017

    2017

  33. [34]

    Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)

    Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668–2677. PMLR, 2018

    2018

  34. [35]

    Concept bottleneck models

    Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Hal Daumé III and Aarti Singh, editors,Proceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Research, pages 5338–5348. PMLR, 13–18 Jul 2020

    2020

  35. [36]

    Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024. URLhttps://arxiv.org/abs/2408.05147

    Pith/arXiv arXiv 2024

  36. [37]

    Sparse autoencoders reveal selective remapping of visual concepts during adaptation, 2025

    Hyesu Lim, Jinho Choi, Jaegul Choo, and Steffen Schneider. Sparse autoencoders reveal selective remapping of visual concepts during adaptation, 2025. URLhttps://arxiv.org/abs/2412.05276

    Pith/arXiv arXiv 2025

  37. [38]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. InEuropean Conference on Computer Vision (ECCV), 2014

    2014

  38. [39]

    Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivo...

  39. [40]

    Sheng Liu, Haotian Ye, Lei Xing, and James Y. Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering.ArXiv, abs/2311.06668, 2023. URL https://api.semanticscholar.org/CorpusID:265149781

    Pith/arXiv arXiv 2023

  40. [41]

    A unified approach to interpreting model predictions

    Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. InAdvances in neural information processing systems 30, 2017

    2017

  41. [42]

    Calm: A causality-guided framework for generating local and global model explanations

    Divyanshu Mahajan, Chenhao Tan, and Matthew Turek. Calm: A causality-guided framework for generating local and global model explanations. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1215–1224, 2021

    2021

  42. [43]

    Promises and pitfalls of black-box concept learning models.ArXiv, abs/2106.13314, 2021

    Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models.ArXiv, abs/2106.13314, 2021. URL https://api.semanticscholar. org/CorpusID:235652059

    Pith/arXiv arXiv 2021

  43. [44]

    Detecting high-stakes interactions with activation probes.ArXiv, abs/2506.10805, 2025

    Alex McKenzie, Urja Pawar, Phil Blandfort, William Bankes, David Krueger, Ekdeep Singh Lubana, and Dmitrii Krasheninnikov. Detecting high-stakes interactions with activation probes.ArXiv, abs/2506.10805, 2025. URLhttps://api.semanticscholar.org/CorpusID:279318482. 13

    arXiv 2025

  44. [45]

    Linguistic regularities in continuous space word representations

    Tomas Mikolov, Wen tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. InNorth American Chapter of the Association for Computational Linguistics, 2013. URL https://api.semanticscholar.org/CorpusID:7478738

    2013

  45. [46]

    Springer Nature Switzerland, 2023

    Georgii Mikriukov, Gesina Schwalbe, Christian Hellert, and Korinna Bade.Evaluating the Stability of Semantic Concept Representations in CNNs for Robust Explainability, page 499–524. Springer Nature Switzerland, 2023. ISBN 9783031440670. doi: 10.1007/978-3-031-44067-0_26. URL http://dx.doi. org/10.1007/978-3-031-44067-0_26

    work page doi:10.1007/978-3-031-44067-0_26 2023

  46. [47]

    Large language models: A survey, 2025

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey, 2025. URL https://arxiv.org/abs/2402.06196

    Pith/arXiv arXiv 2025

  47. [48]

    Fraser, and Svetlana Kiritchenko

    Isar Nejadgholi, Esma Balkır, Kathleen C. Fraser, and Svetlana Kiritchenko. Towards procedural fairness: Uncovering biases in how a toxic language classifier uses sentiment information, 2022. URL https://arxiv.org/abs/2210.10689

    Pith/arXiv arXiv 2022

  48. [49]

    Alison Noble, and Yarin Gal

    Angus Nicolson, Lisa Schut, J. Alison Noble, and Yarin Gal. Explaining explainability: Recommenda- tions for effective use of concept activation vectors, 2025. URLhttps://arxiv.org/abs/2404.03713

    Pith/arXiv arXiv 2025

  49. [50]

    Zoom in: An introduction to circuits.Distill, 2020

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits.Distill, 2020. doi: 10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in

    doi:10.23915/distill.00024.001 2020

  50. [51]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. URL https://openai.com/index/gpt-4o-system-card/ . Model documentation and safety evaluation

    2024

  51. [52]

    isarcasm: A dataset of intended sarcasm

    Silviu Oprea and Walid Magdy. isarcasm: A dataset of intended sarcasm. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020

    2020

  52. [53]

    Disentangling neuron representations with concept vectors

    Laura O’Mahony, Vincent Andrearczyk, Henning Müller, and Mara Graziani. Disentangling neuron representations with concept vectors. In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3770–3775, 2023. doi: 10.1109/CVPRW59228.2023.00390

    arXiv 2023

  53. [54]

    MacArthur, and Christopher R

    Enrico Parisini, Tapabrata Chakraborti, Chris Harbron, Ben D. MacArthur, and Christopher R. S. Banerji. Leakage and interpretability in concept-based models, 2025. URL https://arxiv.org/abs/2504. 14094

    2025

  54. [55]

    Rise: Randomized input sampling for explanation of black-box models

    Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. InProceedings of the British Machine Vision Conference (BMVC), 2018

    2018

  55. [56]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https://arxiv.org/abs/ 2103.00020

    Pith/arXiv arXiv 2021

  56. [57]

    why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier. InProceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016

    2016

  57. [58]

    Steering llama 2 via contrastive activation addition.ArXiv, abs/2312.06681, 2023

    Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition.ArXiv, abs/2312.06681, 2023. URL https://api. semanticscholar.org/CorpusID:266174252

    Pith/arXiv arXiv 2023

  58. [59]

    Roboflow100-vl: A multi-domain object detection benchmark for vision-language models

    Peter Robicheaux, Matvei Popov, Anish Madan, Isaac Robinson, Joseph Nelson, Deva Ramanan, and Neehar Peri. Roboflow100-vl: A multi-domain object detection benchmark for vision-language models. ArXiv, abs/2505.20612, 2025. URLhttps://api.semanticscholar.org/CorpusID:278910603

    arXiv 2025

  59. [60]

    A conversation with bing’s chatbot left me deeply unsettled.The New York Times

    Kevin Roose. A conversation with bing’s chatbot left me deeply unsettled.The New York Times. URL https://www.nytimes.com/2023/02/16/technology/bing-sydney-microsoft-ai-chatbot.html. 14

    2023

  60. [61]

    Friedrich

    Johannes Rückert, Asma Ben Abacha, Alba Garcia Seco de Herrera, Louise Bloch, Raphael Brüngel, Ahmad Idrissi-Yaghir, Henning Schäfer, Henning Müller, and Christoph M. Friedrich. Overview of imageclefmedical 2023 – caption prediction and concept detection. InCLEF 2023: Conference and Labs of the Evaluation Forum, September 2023

    2023

  61. [62]

    Large language models encode semantics in low-dimensional linear subspaces, 2025

    Baturay Saglam, Paul Kassianik, Blaine Nelson, Sajana Weerawardhena, Yaron Singer, and Amin Karbasi. Large language models encode semantics in low-dimensional linear subspaces, 2025. URL https://arxiv.org/abs/2507.09709

    arXiv 2025

  62. [63]

    Visual-tcav: Concept-based attribution and saliency maps for post-hoc explainability in image classification.ArXiv, abs/2411.05698,

    Antonio De Santis, Riccardo Campi, Matteo Bianchi, and Marco Brambilla. Visual-tcav: Concept-based attribution and saliency maps for post-hoc explainability in image classification.ArXiv, abs/2411.05698,

    Pith/arXiv arXiv

  63. [64]

    LAION-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text mod...

    2022

  64. [65]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017

    2017

  65. [66]

    Turner, and Luis Espinosa-Anke

    Zara Siddique, Liam D. Turner, and Luis Espinosa-Anke. Dialz: A python toolkit for steering vectors,

  66. [67]

    Full-gradient representation for neural network visualization

    Suraj Srinivas and François Fleuret. Full-gradient representation for neural network visualization. In Advances in Neural Information Processing Systems 32, 2019

    2019

  67. [68]

    Towards compositionality in concept learning.ArXiv, abs/2406.18534, 2024

    Adam Stein, Aaditya Naik, Yinjun Wu, Mayur Naik, and Eric Wong. Towards compositionality in concept learning.ArXiv, abs/2406.18534, 2024

    Pith/arXiv arXiv 2024

  68. [69]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319–3328. PMLR, 2017

    2017

  69. [70]

    URLhttps://arxiv.org/abs/2505.06262

    Pith/arXiv arXiv

  70. [71]

    Pooling and attention: What are effective designs for llm-based embedding models?, 2024

    Yixuan Tang and Yi Yang. Pooling and attention: What are effective designs for llm-based embedding models?, 2024. URLhttps://arxiv.org/abs/2409.02727

    Pith/arXiv arXiv 2024

  71. [72]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin...

    Pith/arXiv arXiv 2024

  72. [73]

    Investigating task-specific prompts and sparse autoencoders for activation monitoring, 2025

    Henk Tillman and Dan Mossing. Investigating task-specific prompts and sparse autoencoders for activation monitoring, 2025. URLhttps://arxiv.org/abs/2504.20271

    Pith/arXiv arXiv 2025

  73. [74]

    From noise to narrative: Tracing the origins of hallucinations in transformers, 2025

    Praneet Suresh, Jack Stanley, Sonia Joseph, Luca Scimeca, and Danilo Bzdok. From noise to narrative: Tracing the origins of hallucinations in transformers, 2025. URL https://arxiv.org/abs/2509.06938

    arXiv 2025

  74. [75]

    Disentangled representation learning,

    Xin Wang, Hong Chen, Si’ao Tang, Zihao Wu, and Wenwu Zhu. Disentangled representation learning,

  75. [76]

    Manning, and Christopher Potts

    Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. Axbench: Steering llms? even simple baselines outperform sparse autoencoders, 2025. URLhttps://arxiv.org/abs/2501.17148

    Pith/arXiv arXiv 2025

  76. [77]

    Discovering fine-grained visual-concept relations by disentangled optimal transport concept bottleneck models, 2025

    Yan Xie, Zequn Zeng, Hao Zhang, Yucheng Ding, Yi Wang, Zhengjue Wang, Bo Chen, and Hongwei Liu. Discovering fine-grained visual-concept relations by disentangled optimal transport concept bottleneck models, 2025. URLhttps://arxiv.org/abs/2505.07209

    Pith/arXiv arXiv 2025

  77. [78]

    Vazquez, Ulisse Mini, and Monte MacDiarmid

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, 2024. URL https://arxiv. org/abs/2308.10248

    Pith/arXiv arXiv 2024

  78. [79]

    Unmasking and quantifying racial bias of large language models in medical report generation.Communications Medicine, 4(1), September 2024

    Yifan Yang, Xiaoyu Liu, Qiao Jin, Furong Huang, and Zhiyong Lu. Unmasking and quantifying racial bias of large language models in medical report generation.Communications Medicine, 4(1), September 2024. ISSN 2730-664X. doi: 10.1038/s43856-024-00601-z. URL http://dx.doi.org/10. 1038/s43856-024-00601-z

    work page doi:10.1038/s43856-024-00601-z 2024

  79. [80]

    URLhttps://arxiv.org/abs/2211.11695

    Pith/arXiv arXiv

  80. [81]

    Latent concept-based explanation of nlp models.ArXiv, abs/2404.12545, 2024

    Xuemin Yu, Fahim Dalvi, Nadir Durrani, and Hassan Sajjad. Latent concept-based explanation of nlp models.ArXiv, abs/2404.12545, 2024. URL https://api.semanticscholar.org/CorpusID: 269282778

    Pith/arXiv arXiv 2024

Showing first 80 references.