Pith. sign in

REVIEW 17 cited by

NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.14561 v4 pith:KFRPDIYD submitted 2024-07-18 cs.LG cs.AI

Jaden Fiotto-Kaufman , Alexander R. Loftus , Eric Todd , Jannik Brinkmann , Koyena Pal , Dmitrii Troitskii , Michael Ripa , Adam Belfki
show 12 more authors
Can Rager Caden Juang Aaron Mueller Samuel Marks Arnab Sen Sharma Francesca Lucchetti Nikhil Prakash Carla Brodley Arjun Guha Jonathan Bell Byron C. Wallace David Bau
This is my paper · ORCID
classification cs.LGcs.AI
keywords nnsightmodelsinternalsndifaccessconductdeepdesign
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce NNsight and NDIF, technologies that work in tandem to enable scientific study of the representations and computations learned by very large neural networks. NNsight is an open-source system that extends PyTorch to introduce deferred remote execution. The National Deep Inference Fabric (NDIF) is a scalable inference service that executes NNsight requests, allowing users to share GPU resources and pretrained models. These technologies are enabled by the Intervention Graph, an architecture developed to decouple experimental design from model runtime. Together, this framework provides transparent and efficient access to the internals of deep neural networks such as very large language models (LLMs) without imposing the cost or complexity of hosting customized models individually. We conduct a quantitative survey of the machine learning literature that reveals a growing gap in the study of the internals of large-scale AI. We demonstrate the design and use of our framework to address this gap by enabling a range of research methods on huge models. Finally, we conduct benchmarks to compare performance with previous approaches. Code, documentation, and tutorials are available at https://nnsight.net/.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Models

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Entropy-based CoT pruning does not beat random selection; apparent math-task gains come from numeric tokens, not entropy.

  2. ICLR: In-Context Learning of Representations

    cs.CL 2024-12 conditional novelty 7.0 of 10

    As in-context examples grow, Llama-3.1-8B reorganizes its concept representations to mirror the connectivity structure of a graph defined entirely in context.

  3. Activation Probes Surface Code-Security Signals that the Model's Output Misses

    cs.CR 2026-08 conditional novelty 6.0 of 10

    A linear probe read from open-weight LLM activations ranks vulnerable Python functions above their fixes on 61-67% of unseen CVEs, outperforming the same models' prompted verdicts.

  4. Unboxing Diffusion Models for the Arts: Interactive Model Bending and Practice-Based Explainability

    cs.HC 2026-07 conditional novelty 6.0 of 10

    Bending different layers of a diffusion model's UNet produces distinct and fairly consistent visual effects across seeds and prompts, and an interactive ComfyUI tool lets artists explore these effects hands-on.

  5. Revealing Hidden Model Behaviors with Task-Specific Self-Reports

    cs.CL 2026-07 conditional novelty 6.0 of 10

    SAR detects every implanted hidden behavior across eight Qwen3-14B settings and halves IA’s hallucination rate by aligning self-report activations to a contrastive behavior direction under a coherent-English stabilizing cap.

  6. Language Triggers Hijack Language Circuits: A Mechanistic Analysis of Backdoor Behaviors in Large Language Models

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Backdoor triggers in a family of LLMs activate the same attention heads that encode output language, suggesting triggers co-opt existing language circuits rather than forming new ones.

  7. Interpreto: An Explainability Library for Transformers

    cs.CL 2025-12 accept novelty 6.0 of 10

    Interpreto provides a unified, documented Python API for attribution and end-to-end concept-based explanation of HuggingFace classification and generation models.

  8. Insights into a radiology-specialised multimodal large language model with sparse autoencoders

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Applying Matryoshka sparse autoencoders to a radiology-specialised multimodal LLM reveals a minority of interpretable clinical features, while steering them produces unreliable and often off-target report changes.

  9. Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Models

    cs.LG 2025-05 conditional novelty 6.0 of 10

    ITDA dictionaries, built greedily from poorly reconstructed activations and decomposed by matching pursuit, match some SAE reconstruction performance at 100x lower training cost and enable SOTA cross-model layer simil...

  10. Steering Large Language Models for Machine Translation Personalization

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Contrastive steering of sparse autoencoder features personalizes literary machine translation to a target translator's style as well as twenty-shot prompting while keeping inference fast.

  11. Explaining Neural Networks with Reasons

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A new interpretability method computes 'reasons vectors' from neuron activations and measures how strongly each neuron supports propositions about the input, with experiments on MNIST, Adult, and SST2.

  12. Language Models Use Trigonometry to Do Addition

    cs.AI 2025-02 conditional novelty 6.0 of 10

    Three LLMs represent two-digit numbers as generalized helices and appear to compute addition by combining these helices into an answer helix.

  13. Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Models

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Autoregressive language models represent multiple competing readings of garden path sentences via interpretable syntactic features, and do not reuse those features to answer follow-up questions.

  14. InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoders

    q-bio.BM 2024-11 conditional novelty 6.0 of 10

    Sparse autoencoders trained on ESM-2 recover thousands of interpretable features that align with Swiss-Prot concepts and can influence sequence generation.

  15. Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers

    cs.CL 2024-11 conditional novelty 6.0 of 10

    By patching averaged concept representations across languages into transformer models, the authors show the models can still translate and even describe those concepts, supporting the idea of language-agnostic concept...

  16. BlueGlass: A Framework for Composite AI Safety

    cs.AI 2025-07 conditional novelty 5.0 of 10

    BlueGlass provides composite AI safety infrastructure; its case studies on object-detection VLMs reveal dataset trade-offs, a decoder-layer phase transition in probe accuracy, and SAE-discovered concepts including spu...

  17. eDIF: A European Deep Inference Fabric for Remote Interpretability of LLM

    cs.CL 2025-08 conditional novelty 4.0 of 10

    A European, NDIF-compatible cluster for remote LLM interpretability is technically feasible and positively received in a 16-person pilot, though the user-study evidence is thin.

Pith tools