Pith. sign in

REVIEW 4 major objections 5 minor 39 references

Sparse Activation Editing for Reliable Instruction Following in Narratives

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Editing 30 internal sparse features makes frozen LLMs follow narrative constraints up to three times more often, without retraining or labels.

desk verdict A plausible SAE-steering method and a useful new benchmark, but the headline FreeInstruct gains are undercut by tuning on the evaluation set and a missing XSTest table. read the letter →

arxiv 2505.16505 v1 pith:HBBZ2ASJ submitted 2025-05-22 cs.CL cs.AIcs.HC

Runcong Zhao , Chengyu Cao , Qinglin Zhu , Xiucheng Lv , Shun Shao , Lin Gui , Ruifeng Xu , Yulan He This is my paper
classification cs.CLcs.AIcs.HC
keywords sparseautoencodersinstructionfollowingrepresentationengineeringneuroneditingadversarialinstructionsnarrativebenchmarkBayesianoptimizationtraining-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Concise-SAE, a training-free method for making LLMs follow instructions when a user tries to bypass an established narrative constraint, such as introducing supernatural elements into a realistic mystery. It claims that a sparse autoencoder can localize a small set of internal neurons that encode the instruction, and that injecting an optimized shift into those neurons at inference time makes the model adhere to the constraint while avoiding both blind compliance and robotic refusal. The method is validated on a new 1,212-example benchmark, FreeInstruct, built by human annotators around adversarial and normal user inputs; it reports instruction-following rates of 0.860, 0.887, and 0.533 on Llama-3.1-8B, Gemma-2-9B, and Gemma-2-2B, roughly 2.4 to 3 times the no-control baseline. The authors argue that existing instruction-following benchmarks miss the "bypass-scenario-constraints" failure mode, so they contribute both the benchmark and the editing recipe.

What carries the argument

The engine is attention-anchored sparse-activation editing. A short keyword token that summarizes the instruction is appended to the input; in a decoder-only transformer the residual of that final token aggregates context through attention, and an SAE maps it to a high-dimensional sparse code whose dimensions are meant to be monosemantic features. The attribution statistic $\Delta_p$ averages, across LLM-generated follow/violate rewrites, whether neuron p's thresholded activation flips, and the most supportive and most violating neurons define a 2k-dimensional steering subspace. A Gaussian-process Bayesian optimizer then chooses the edit coefficients, balancing instruction adherence, refusal avoidance, and output quality as scored by the base model itself; at inference the learned sparse shift is added to the code before decoding.

What would settle it

Take the top-30 steering set for an instruction, then rerun the exact same edit on a held-out set of FreeInstruct stories with 30 randomly chosen neurons from the same SAE layer; if random-neuron edits produce a comparable IFR gain, the attribution step is not the cause of the improvement. A complementary check is to compute the steering vector on one narrative genre and apply it to another narrative with the same instruction—if the IFR gain vanishes, the selected neurons encode rewrite style rather than the instruction.

Watch

Extended reading notes

Core claim

Concise-SAE shows that instruction-following behavior can be edited through a small set of sparse autoencoder features found without labels. Given an instruction, the model rewrites stories to follow and to violate it; a keyword token summarizing the instruction is appended to each input, its residual representation is SAE-encoded, and neurons are ranked by how consistently their thresholded activations separate follow from violate outputs. The central empirical discovery is that supportive and opposing neurons form approximately orthogonal subspaces, so both must be edited; with the top-k from each side and Bayesian-optimized coefficients, the method lifts FreeInstruct IFR from 0.340 to 0.860 on Llama-3.1-8B, 0.307 to 0.887 on Gemma-2-9B, and 0.227 to 0.533 on Gemma-2-2B without sacrificing response rate or output quality.

Load-bearing premise

The load-bearing premise is that the sparse neurons selected by thresholded differences between follow and violate rewrites are the actual causal controllers of instruction-following, rather than merely correlating with stylistic differences in the generated examples.

Editorial extensions

If this is right

  • Instruction adherence can be improved at inference time for any LLM with an available public SAE, with no supervised data and no weight updates.
  • The orthogonality result means representation-steering methods should treat "follow" and "violate" as separate subspaces instead of assuming a single contrastive direction.
  • FreeInstruct-style evaluation, with adversarial inputs embedded in narrative worlds, should be added to instruction-following benchmarks because current benchmarks miss the bypass-constraint failure mode.
  • Automatic magnitude optimization removes the fixed-strength hyperparameter that prior editing methods require; moderate edits work and over-editing causes repetition or evasion.
  • Concise-SAE also improves IFR on WildGuard and prompt-injection tasks, suggesting the mechanism is not specific to the new benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A likely generalization, not pursued in the paper: the same keyword-anchored contrastive localization could steer style, tone, or factual attributes, since any attribute that can be summarized by a short phrase can seed the follow/violate rewrites.
  • The causal claim is bounded by the rewrite generator: if the follow/violate rewrites differ mainly in style rather than in instruction-relevant content, the selected neurons will encode that surface difference; a stability check across different generator prompts would test this.
  • Because the reward uses the base model's own judgments of compliance and quality, the optimization could exploit self-assessment biases; an independent judge held out from optimization would reveal whether the gains are genuine.
  • The small 30-neuron subspace suggests similar sparse edits might correct other behavioral failures such as sycophancy or over-refusal, but the paper does not test those.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Concise-SAE, a training-free representation-editing method that uses sparse autoencoder (SAE) latents to identify instruction-relevant neurons from contrastive LLM-generated follow/violate story pairs, then steers a small set of supportive and opposing neurons with coefficients found by Bayesian optimization. It also introduces FreeInstruct, a 1,212-example benchmark of instruction following in narrative-rich settings, and reports large IFR gains on it (0.860 for Llama-3.1-8B, 0.887 for Gemma-2-9B, 0.533 for Gemma-2-2B) as well as improvements on WildGuard and Prompt Injection. The method is unsupervised and label-free for neuron localization, relying on keyword-token semantic aggregation plus an exponential noise-suppression argument in Section 3.1, and on GPT-4o-based evaluation for the reported metrics.

Significance. If the reported results are out-of-sample, this is a useful contribution: the paper offers a training-free, label-free way to localize instruction-following features with publicly available SAEs and to edit them with a compact set of coefficients. FreeInstruct addresses a real evaluation gap, the annotation protocol is described in unusual detail, code and data are promised, and the limitations section is candid about SAE availability, self-evaluation cost, and evaluator bias. The theoretical noise-suppression bound in Section 3.1 is a concrete, falsifiable claim, and the bidirectional supportive/opposing steering idea is well motivated by the cosine-similarity analysis in Section 5.2. However, the headline empirical claim is not currently established because the paper does not show that its hyperparameters were selected without access to the FreeInstruct test examples, and the self-evaluated optimization reward introduces an additional circularity risk.

major comments (4)
  1. [Section 5.1, Section 3.2, Appendix A.3, Appendix C.2] The paper never defines a train/validation/test split for FreeInstruct, and all hyperparameters appear to be selected on the same 1,212 examples used to report Table 1. Section 3.2 runs Bayesian optimization on 'a fixed minibatch of examples' without stating its size or source; Appendix A.3 chooses k=15 using Figure 6, which plots FreeInstruct IFR as a function of k; and Appendix C.2 chooses layer 15 using Table 5, which is also expressed in FreeInstruct IFR. If the BO minibatch and the k/layer ablations are drawn from the same benchmark examples, the reported IFR numbers are in-sample and the comparison against baselines that receive no equivalent tuning is unfair. The revision must provide an explicit split, fix all hyperparameters before seeing the test set, state the BO minibatch size and origin, and report test-set metrics from that protocol.
  2. [Section 3.2] The optimization reward R(lambda) is computed by the base LLM itself: r_inst, r_ref, and r_qual are all 'computed by the base LLM', while the reported FreeInstruct metrics use GPT-4o. This creates a risk that BO exploits the base model's self-scoring weaknesses, producing coefficients that maximize the self-reward but not the GPT-4o-evaluated IFR. Since the same self-evaluated reward also sits inside the hyperparameter-selection loop, the circularity is not limited to the final metric. Please report agreement between the self-reward and GPT-4o on a validation set, and test whether optimizing with an independent reward (e.g., GPT-4o as reward on the same minibatch) changes the conclusions.
  3. [Section 5.2, Appendix A.3] There are no error bars or repeated-run statistics for any of the headline numbers. Appendix A.3 says greedy decoding makes outputs deterministic and each query is run once, but determinism does not remove variance from the choice of BO minibatch, the random initial steering vectors drawn from N(0,1), the selection of k and layer, or the particular FreeInstruct examples used. Table 1 reports single numbers, and several differences that support the SOTA claim are modest (e.g., 0.860 vs. 0.787 for ICV on Llama-3.1-8B). Please report means and standard deviations over at least 3-5 independent optimization runs, or bootstrapped confidence intervals over the benchmark examples.
  4. [Section 3.2 vs. Appendix A.1] The optimization dimension is inconsistent. Section 3.2 defines a steering vector over 2k selected neurons, lambda = sum_{ell=1}^{2k} lambda_ell e_{p_ell}, which with k=15 gives a 30-dimensional search space, consistent with the text saying 'the resulting 2k-dimensional space'. Appendix A.1 instead says the steering vector has non-zero entries only at the k neuron positions and writes lambda = sum_{ell=1}^k lambda_ell e_{p_ell}, calling it a k-dimensional subspace. This contradiction affects the number of initial samples, the GP kernel length scales, and the interpretation of the bidirectional ablation. Please correct the formula and state the exact dimension used in the reported experiments.
minor comments (5)
  1. [Equation (1)] The quantity Delta is written as Delta_{pp} in Equation (1); the repeated subscript appears to be a typo and should be Delta_p.
  2. [Figure 5 caption] The caption contains a stray fragment at the end: 'Prior work uses fixed strength (a hyperparameter); ours learns via optimisation. ours .' The trailing 'ours .' should be removed.
  3. [Table 2] The rows labeled 'First irrelevant', 'Middle irrelevant', and 'Last irrelevant' are not self-explanatory; please clarify that these refer to the position of the keyword token in the input, and define what 'irrelevant' versus 'relevant' means for the keyword's semantic content.
  4. [Section 4] The benchmark name is typeset inconsistently as FREEINSTRUCT, FreeInstruct, and FreeInstruct across the abstract, body, and tables; please standardize the capitalization.
  5. [References] The reference to An et al. (2024) cites arXiv:2502.04153, which is a 2025 preprint; please verify the year and update the citation to avoid a mismatch with the in-text year.

Circularity Check

1 steps flagged · score 6.0 of 10

FreeInstruct IFR is tuned on the benchmark itself: BO coefficients and k are selected with no held-out split, so the headline gains are fitted, not predicted.

  1. fitted input called prediction [Section 3.2 (Representation Steering); Appendix A.3 (Hyperparameters & Setup); Section 5.1 (Experimental Setup)]
    "A fixed minibatch of examples is used throughout the entire optimisation process, and R(λ) is self-evaluated by the LLM at each iteration. ... we set k= 15 by selecting the top 15 neurons that most strongly support the instruction and the top 15 that most consistently violate it, based on the attribution metric defined in Equation 1. ... As shown in Figure 6, performance improves from k=5 to k=15, but plateaus or slightly declines at k=20."

    The headline FreeInstruct IFR (0.860/0.887/0.533) is reported after Bayesian optimisation of λ on 'a fixed minibatch of examples' using R(λ), whose r_inst term is 'a binary score indicating whether the response follows the target instruction t', and after selecting k=15 from Figure 6's performance-vs-k curve. Section 5.1 defines no train/validation/test split and only says the models 'are primarily evaluated on our proposed dataset FREEINSTRUCT.' Because the paper never shows the minibatch or the k-selection ablations to be disjoint from the 1,212 FreeInstruct examples, the reported IFR is an optimized value on the evaluation benchmark rather than an out-of-sample prediction; the SOTA gain is a fitted input renamed as a measured result.

full rationale

The central FreeInstruct result is not a self-contained derivation: the coefficients λ and the neuron count k are tuned via a self-evaluated reward on a fixed minibatch and a performance-vs-k curve, and no held-out split is described before the same benchmark is used to report IFR. This makes the headline 2.3x–3x gains in Table 1 a re-measurement of the optimized objective rather than an independent prediction. The circularity is partial rather than total: WildGuard, Prompt Injection, and XSTest are external benchmarks not used for tuning and provide some independent support, and the released SAEs (Gemma Scope, Llama Scope) are external artifacts with no self-citation chain. No load-bearing self-citation or imported uniqueness theorem is present. The paper's own Limitations note the cost of self-evaluation queries and GPT-4o evaluator bias, but not the missing train/test separation, which is the specific reduction identified here.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claim depends on two kinds of unpaid inputs: hand-picked or benchmark-selected hyperparameters (k, layer, threshold, keyword, and optimized coefficients), and assumptions that public SAEs are monosemantic, that keyword-token aggregation captures instruction signal, and that the self-evaluated reward is a valid optimization target. The paper does not disclose exact values for the threshold, the final steering vectors, or the Bayesian optimization minibatch, which limits the completeness of this ledger.

free parameters (5)
  • neuron count k = k=15 per direction (30 total)
    Selected by running the method on FreeInstruct and picking the best value in Figure 6; not set a priori.
  • edit layer L = layer 15
    Selected by ablating layers on FreeInstruct in Table 5; the best layer is chosen on the same benchmark used for reporting.
  • selection threshold tau = not reported
    Used in Equation 1 for neuron selection, but its numerical value is never given in the paper.
  • steering coefficients lambda* = not reported (30 per instruction)
    Optimized by Bayesian optimization on a fixed minibatch against a self-evaluated reward; the resulting vectors are not released, so edits cannot be reproduced without rerunning the optimizer.
  • keyword summary word x* = human-chosen, e.g. 'realistic'
    A human chooses the keyword that summarizes each instruction; semantic choice materially changes performance according to Table 2.
assumptions (5)
  • domain assumption Released SAEs provide approximately monosemantic sparse features that support causal editing.
    Section 2 and Section 3.1 rely on publicly available GemmaScope and LlamaScope SAEs; if features are not monosemantic or causally editable, neuron selection does not control behavior.
  • ad hoc to paper The residual-stream representation of an appended keyword token aggregates the instruction signal across the context.
    Section 3.1 models the keyword residual as an attention-weighted sum and then treats it as a summary of instruction behavior.
  • ad hoc to paper Non-target SAE activations are independent sub-Gaussian noise, justifying the exponential noise-suppression bound.
    Formalized in Section 3.1; the bound's validity depends on this modeling assumption, which is not empirically verified.
  • domain assumption The fixed minibatch used for Bayesian optimization is representative and does not overlap the evaluation set.
    Section 3.2 optimizes coefficients on a fixed minibatch and the reported numbers come from the full benchmark, but no train/eval split or overlap analysis is documented.
  • domain assumption The base LLM's self-evaluated reward is a valid proxy for true instruction adherence.
    The reward components in Section 3.2 are computed by the same model being edited, which risks reward hacking; the paper does not analyze this.
invented entities (1)
  • Supportive versus opposing instruction-neuron subspaces
    purpose: Central steering construct: top-k neurons that follow the instruction and top-k that violate it are edited jointly because they are claimed to be orthogonal subspaces.
    Figure 4 and Appendix C.1 provide internal evidence only; no independent or external verification is given that these subspaces exist outside the selected examples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sparse Activation Editing for Reliable Instruction Following in Narratives." pith.science (2026). https://pith.science/paper/HBBZ2ASJ

@misc{pith2026250516505,
  author       = {Pith},
  title        = {Pith review of: Sparse Activation Editing for Reliable Instruction Following in Narratives},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HBBZ2ASJ}},
  note         = {Machine review of arXiv:2505.16505}
}
read the original abstract

Complex narrative contexts often challenge language models' ability to follow instructions, and existing benchmarks fail to capture these difficulties. To address this, we propose Concise-SAE, a training-free framework that improves instruction following by identifying and editing instruction-relevant neurons using only natural language instructions, without requiring labelled data. To thoroughly evaluate our method, we introduce FreeInstruct, a diverse and realistic benchmark of 1,212 examples that highlights the challenges of instruction following in narrative-rich settings. While initially motivated by complex narratives, Concise-SAE demonstrates state-of-the-art instruction adherence across varied tasks without compromising generation quality.

Figures

Figures reproduced from arXiv: 2505.16505 by the authors.

Figure 1
Figure 1. Examples of user inputs that deviate from in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Contrastive neuron identification. Given an instruction, we prompt the LLM to generate a pair of stories—one that follows the instruction and one that violates it. A keyword token (e.g., “realistic”) summarising the instruction is appended to each input, and its residual representation h⋆ is extracted from a target LLM layer. These are encoded via an SAE to obtain sparse vectors z⋆, which are used to rank neurons ba… view at source ↗
Figure 3
Figure 3. Overview of the FREEINSTRUCT data construction process. The boxed components represent the final structure of each FREEINSTRUCT example: (story, normal input, adversarial input, expected output). selection of new candidates using EI. This process continues until convergence, yielding the optimal coefficients λ ⋆ = arg max R. Further theoretical foundations and implementation details are pro￾vided in Appendix A.1. 4 … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Pairwise cosine similarity between neurons se [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Edit Direction and Strength. (a) Single￾direction edits miss complementary control from oppos￾ing neurons in distinct subspaces. (b) Excessive strength degrades output; our method learns it automatically. Effectiveness of Coefficient Optimisation We compare three editi…
Figure 6
Figure 6. Figure 6: Performance across different numbers of se [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 11 canonical work pages

  1. [1]

    Harsh Agrawal, Aditya Mishra, Manish Gupta, and Mausam. 2023. Multimodal persona based generation of comic dialogs. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14150--14164, Toronto, Canada. Association for Computational Linguistics

    work page 2023

  2. [2]

    Kaikai An, Li Sheng, Ganqu Cui, Shuzheng Si, Ning Ding, Yu Cheng, and Baobao Chang. 2024. https://doi.org/10.48550/arXiv.2502.04153 Ultraif: Advancing instruction following from the wild . Preprint, arXiv:2502.04153. https://arxiv.org/abs/2502.04153

    work page doi:10.48550/arxiv.2502.04153 2024

  3. [3]

    Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, David Molnar, Spencer Whitman, and Joshua Saxe. 2024. https://arxiv.org/abs/2404.13161 Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models . Preprint, arXiv:2404.13161

    arXiv 2024

  4. [4]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Ch...

    2023

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    work page 2020

  6. [6]

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. https://arxiv.org/abs/2309.08600 Sparse autoencoders find highly interpretable features in language models . Preprint, arXiv:2309.08600

    arXiv 2023

  7. [7]

    Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. 2025. https://github.com/QwenLM/AutoIF Self-play with execution feedback: Improving instruction-following capabilities of large language models . In Proceedings of the Twelfth International Conference on Learning Representations (ICLR). Spotlight

    work page 2025

  8. [8]

    Peter I. Frazier. 2018. https://arxiv.org/abs/1807.02811 A tutorial on bayesian optimization . Preprint, arXiv:1807.02811

    arXiv 2018

Show all 39 references
  1. [9]

    Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. 2024. https://arxiv.org/abs/2406.18495 Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms . Preprint, arXiv:2406.18495

    2024 arXiv

  2. [10]

    Nikolaus Hansen. 2016. https://doi.org/10.48550/arXiv.1604.00772 The cma evolution strategy: A tutorial . arXiv preprint arXiv:1604.00772. Version 2, updated 2023

    2016 doi arXiv

  3. [11]

    Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. 2024. https://doi.org/10.48550/arXiv.2410.20526 Llama scope: Extracting millions of features from llama-3.1-8b with sp...

    2024 doi arXiv

  4. [12]

    Zirui He, Haiyan Zhao, Yiran Qiao, Fan Yang, Ali Payani, Jing Ma, and Mengnan Du. 2025. Saif: A sparse autoencoder framework for interpreting and steering instruction following of language models. arXiv preprint arXiv:2502.11356

    2025 arXiv

  5. [13]

    Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.189 Lion: Adversarial distillation of proprietary large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages ...

    2023 doi

  6. [14]

    Yuxin Jiang, Yufei Wang, Chuhan Wu, Wanjun Zhong, Xingshan Zeng, Jiahui Gao, Liangyou Li, Xin Jiang, Lifeng Shang, Ruiming Tang, Qun Liu, and Wei Wang. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.258 Learning to edit: Aligning LLMs with knowledge editing . In Proceeding...

    2024 doi

  7. [15]

    Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.257 Followbench: A multi-level fine-grained constraints following benchmark for large language models ....

    2024 doi

  8. [16]

    Jiazheng Li, Artem Bobrov, David West, Cesare Aloisi, and Yulan He. 2025. https://doi.org/10.1609/aaai.v39i28.35358 An automated explainable educational assessment system built on llms . In Proceedings of the AAAI Conference on Artificial Intelligence: Demonstration Track, vol...

    2025 doi

  9. [17]

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. https://doi.org/10.48550/arXiv.2408.05147 Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2 . ...

    2024 doi arXiv

  10. [18]

    Sheng Liu, Haotian Ye, Lei Xing, and James Zou. 2024 a . In-context vectors: making in-context learning more effective and controllable through latent space steering. In Proceedings of the 41st International Conference on Machine Learning, volume 238 of Proceedings of Machine ...

    2024

  11. [19]

    Sheng Liu, Haotian Ye, Lei Xing, and James Zou. 2024 b . https://arxiv.org/abs/2311.06668 In-context vectors: Making in context learning more effective and controllable through latent space steering . Preprint, arXiv:2311.06668

    2024 arXiv

  12. [20]

    Junru Lu, Jiazheng Li, Guodong Shen, Lin Gui, Siyu An, Yulan He, Di Yin, and Xing Sun. 2025. https://doi.org/10.48550/arXiv.2502.11387 Rolemrc: A fine-grained composite benchmark for role-playing and instruction-following . arXiv preprint arXiv:2502.11387

    2025 doi arXiv

  13. [21]

    Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller

    Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2025. https://arxiv.org/abs/2403.19647 Sparse feature circuits: Discovering and editing interpretable causal graphs in language models . Preprint, arXiv:2403.19647

    2025 arXiv

  14. [22]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2023. https://arxiv.org/abs/2202.05262 Locating and editing factual associations in gpt . Preprint, arXiv:2202.05262

    2023 arXiv

  15. [23]

    Meta. 2024. https://ai.meta.com/blog/meta-llama-3/ Introducing meta llama3: The most capable openly available llm to date

    2024

  16. [24]

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kam...

    2022

  17. [25]

    Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S

    Joon Sung Park, Carolyn Q. Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S. Bernstein. 2024. https://doi.org/10.48550/arXiv.2411.10109 Generative agent simulations of 1,000 people . arXiv preprint arXiv:2411.10109

    2024 doi arXiv

  18. [26]

    Rush, and Thomas Wolf

    Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. 2023. https://huggingface.co/datasets/HuggingFaceH4/no_robots No robots

    2023

  19. [27]

    Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. https://doi.org/10.18653/v1/2020.acl-main.647 Null it out: Guarding protected attributes by iterative nullspace projection . In Proceedings of the 58th Annual Meeting of the Association for Com...

    2020 doi

  20. [28]

    Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. https://arxiv.org/abs/2308.01263 Xstest: A test suite for identifying exaggerated safety behaviours in large language models . Preprint, arXiv:2308.01263

    2024 arXiv

  21. [29]

    Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. https://aclanthology.org/2024.acl-long.309/ Language-specific neurons: The key to multilingual capabilities in large language models . In Proceedings of the 62nd A...

    2024

  22. [30]

    Gemma Team. 2024. https://arxiv.org/abs/2408.00118 Gemma 2: Improving open language models at a practical size . Preprint, arXiv:2408.00118

    2024 arXiv

  23. [31]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua ...

    2024

  24. [32]

    Vazquez, Ulisse Mini, and Monte MacDiarmid

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2024. https://arxiv.org/abs/2308.10248 Steering language models with activation engineering . Preprint, arXiv:2308.10248

    2024 arXiv

  25. [33]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. https://openreview.net/forum?id=gEZrGCozdqR Finetuned language models are zero-shot learners . In The Tenth International Conference on Learning Repr...

    2022

  26. [34]

    John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. 2023. Intercode: Standardizing and benchmarking interactive coding with execution feedback. In Proceedings of the Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track

    2023

  27. [35]

    Yu Zhao, Alessio Devoto, Giwon Hong, Xiaotang Du, Aryo Pradipta Gema, Hongru Wang, Xuanli He, Kam-Fai Wong, and Pasquale Minervini. 2025. Steering knowledge selection behaviours in llms via sae-based representation engineering. arXiv preprint arXiv:2410.15999

    2025 arXiv

  28. [36]

    Qinglin Zhu, Runcong Zhao, Bin Liang, Jinhua Du, Lin Gui, and Yulan He. 2024. https://doi.org/10.48550/arXiv.2404.17662 Player*: Enhancing llm-based multi-agent communication and interaction in murder mystery games . arXiv preprint arXiv:2404.17662

    2024 doi arXiv

  29. [37]

    Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico...

    2025 arXiv

  30. [38]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.