{"id":"93312908-a300-4f8d-b88a-8b8c60834e25","arxiv_id":"2611.17058","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Internal neural representations can be treated as an interaction material, and a four-dimensional design space shows most current interfaces target expert auditors rather than lay users.","lead":"This paper introduces Neural Representation Interaction (NRI), a framework for letting people inspect, steer, and edit the internal representations of neural networks instead of only prompting them. It maps existing tools onto a four-dimensional design space and argues that most currently serve expert auditors, with few interfaces built for everyday users.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"NRI's central affordance depends on MI outputs being faithful and lay-readable; the paper concedes both are open (b218-b220) yet builds the design space and applications as if they held, without a validation protocol.","rationale":"The reader's weakest assumption names the same hinge: internal representations must be accessible and meaningful to users through MI tools, and the paper concedes the underlying reliability questions. My stress-test agrees and sharpens the point: because NRI is defined by users interpreting and directing representations, MI fidelity is constitutive of the paradigm, not an implementation detail. The paper is honest in Section 6, and the conditional verdict already reflects that honesty. What remains unresolved is that the design space, the Table 2 mapping, and the speculative applications are built as if the reliability question had been settled, with no protocol for validating whether a user-facing operation actually targets the model's computation. The concrete adversarial-label test would directly probe whether lay users can distinguish faithful MI output from plausible filler; that is the minimal empirical condition for NRI to be more than a relabeling of expert interpretability tools.","tokens_in":23821,"tokens_out":5358,"duration_ms":43831,"concrete_test":"Run a pre-registered adversarial-label study in a TalkTuner-style interface: take a real steering-direction UI on a small open LLM and, for half of the concepts, replace the MI-provided labels with random labels sampled from the same vocabulary; ask lay participants to judge which labels are real by testing each control's visible behavioral effects. If label-discrimination accuracy is near chance, NRI interfaces are operating on plausible descriptions rather than on representations users can genuinely interpret and direct; if participants detect the swap reliably, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that users can 'interpret and direct the neural representations of AI systems' (Section 1) rests on two coupled conditions: (i) MI outputs such as probes, SAEs, and verbalizations correspond to the model's actual computation, and (ii) nonexpert users can read those outputs and act on them without expert training. The paper itself states that not all features are linear, that SAEs 'may not recover ground-truth and human-interpretable features,' and that verbalization 'may risk confabulating descriptions' (b218-b220). This is an acknowledged limitation, but it is load-bearing rather than peripheral: if an SAE label or verbalized description does not track the computation it claims to expose, then an NRI interface is not giving users access to neural representations; it is presenting a plausible-sounding explanation artifact, and 'directing representations' reduces to adjusting a visualization. The paper does not integrate this uncertainty into the framework: the Section 3 design space has no fidelity or validation dimension, the Section 4 mapping to Table 2 supplies no codebook for classifying interfaces or user groups, so the claim that only three systems serve lay users (b144, Table 2 note) is not auditable, and the Section 5 speculative applications assume reliable causal attribution and intervention. TalkTuner's positive user-study result (Section 4.1) is a useful existence proof, but it does not establish the general paradigm.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper introduces Neural Representation Interaction (NRI), defined as “interaction that enables users to interpret and direct the neural representations of AI systems.” It situates NRI alongside GUI, TUI, and living bits, synthesizes actionable mechanistic interpretability methods (probing, steering, verbalization, sparse autoencoders, attribution), and constructs a GMA-based design space with four facets: Interaction, Target, Referent, and Purpose. The authors map ten existing interpretability interfaces onto this space (Table 2) and claim that most serve expert auditors while few support lay users (abstract, Section 4). The paper then presents speculative applications in education (Shakespeare Writing Studio), personal modeling (Latent Mirror), and hiring audits (Causal Auditor), and closes with a limitations section and a sociotechnical risk discussion.","tokens_in":25759,"tokens_out":2494,"duration_ms":117240,"significance":"If the conceptual claim holds, NRI offers HCI a genuinely new paradigm that lets users work with AI computation rather than only through prompts and outputs. The paper's strengths are its careful synthesis of a fragmented MI literature, its explicit and candid enumeration of technical reliability limits (Section 6), and its attempt to give interaction designers a reusable vocabulary (Inspect/Steer/Edit; activations/components/circuits/parameters; concepts/computation/user information/behavior; understanding/sensemaking/decision support/auditing/control). The TalkTuner existence proof is a useful anchor. The paper does not ship new empirical data, but as a conceptual and design-space contribution it is potentially valuable for the HCI community.","major_comments":[{"comment":"The central mapping claim that “most serve expert auditors, while few support lay users” (abstract, b5) and that “only TalkTuner, Neural Transparency, Multi-Turn Neural Transparency fall under interfaces for lay users” (Table 2 note, b144) is not auditable. The paper does not provide a codebook for how systems were selected, how “lay user” versus “expert auditor” was operationalized, or how the four-facet assignments were made and checked (e.g., by independent raters). Since this empirical claim is a headline result, the authors should add an explicit classification procedure and a justification for the system-selection boundary.","section":"Section 4 / Table 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The claim that the four facets form a complete and cross-consistent design space is an ad hoc assumption. The paper describes the GMA procedure (extraction from surveys, affinity mapping) but does not report whether any candidate facets were discarded, how the named four were chosen over alternatives, or any validity check for the cross-consistency of the cells. Without such transparency, the design space is a useful heuristic but the completeness claim should be softened or made testable.","section":"Section 3 (GMA facet selection)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Section 6 openly acknowledges that probes/steering rest on the linear-representation hypothesis, that SAEs “may not recover ground-truth and human-interpretable features,” and that verbalization “may risk confabulating descriptions” (b218–b220). Yet the speculative applications in Section 5, particularly Latent Mirror and Causal Auditor, assume faithful causal attribution and reliable intervention without a validation protocol. The authors should integrate the reliability uncertainty into the design narrative: e.g., propose concrete validation steps (ground-truth feature checks, causal-scrubbing-style tests, user studies measuring downstream trust calibration) or explicitly frame the applications as contingent on future MI reliability.","section":"Section 5 vs. Section 6","authors_acknowledge":"For instance, probes and steering vectors rest on the linear represen- tation hypothesis, yet not all features seem to be linear [37], mo- tivating ongoing work on manifold-based steering [134]; existing steering vectors generalize inconsistently beyond the distribution they were derived from and produce unintended side effects on unrelated behavior [121].","authors_acknowledge_p":0.66},{"comment":"The term “representational bits” is introduced as a core concept of NRI and used rhetorically throughout the paper, but it is never given an operational definition. As written, it conflates activation values, SAE features, and edit targets. A clear formal definition (or an explicit statement that “representational bits” is a rhetoric-level placeholder for MI features) is needed so that claims such as “users can act on representational bits” are meaningful.","section":"Section 1 (representational bits)","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The sentence “There is a large body of evidence that features within LLMs are represented as linear directions” should be balanced with the acknowledged nonlinear-features caveat earlier in the paragraph, so the reader is not left with the impression that linearity is the norm.","section":"Section 2.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The sentence “Parameter have soft incompatibility issues with user information which is often measured with the model activations” contains a grammatical error and an unclear subject; revise for clarity.","section":"Section 3 (cross-consistency)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The Circuit Tracer row lists Target as “Circuits, Components” but the accompanying text in Section 4.2 also refers to tracing circuits and editing activations; please ensure the table cells match the prose exactly.","section":"Table 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"References [81] and [82] are the same ROME paper listed twice with slightly different formatting; merge them into a single entry to avoid duplicate citations.","section":"References","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The visualization paragraph is informative but abruptly placed after the cross-consistency discussion; it might fit better as a short subsection or be explicitly tied to the four-facet table.","section":"Section 3 (Visualizations)","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Stanley,\n\nThis is a serious conceptual paper that does something useful: it names and organizes a design space that HCI researchers have been circling but haven't articulated as a paradigm. The four dimensions (Interaction, Target, Referent, Purpose) are sensible, and the mapping of existing tools onto them is a legitimate synthesis. The paper is honest about its limitations—Section 6 openly lists the reliability problems of probes, SAEs, and verbalization, and the architectural assumptions baked into the framework. That candor is a real strength.\n\nWhat's new is the framing itself: treating internal representations as an interaction material, analogous to how GUI treats pixels and TUI treats physical objects. That's a genuine conceptual contribution, and the paper earns it by situating NRI against prior paradigms and by showing how existing MI interfaces fit (or don't fit) the space.\n\nNow the soft spots. The biggest is that the central mapping claim—that most interfaces serve expert auditors while few support lay users—rests on a curated table (Table 2) with no systematic corpus or codebook. The classification of which interfaces serve lay users is plausible but not auditable. More importantly, the paradigm's viability for lay users depends on MI outputs being both faithful and interpretable without training. The paper concedes both are open problems (b218–b220), yet the design space has no fidelity or validation dimension, and the speculative applications in Section 5 assume reliable causal attribution and intervention. That's not a fatal flaw for a conceptual paper, but it means the paradigm is currently a promissory note rather than a demonstrated interaction mode.\n\nMinor: the GMA methodology is described but not validated—no inter-rater reliability, no systematic extraction protocol. The paper would be stronger with a reproducibility appendix.\n\nWho this is for: HCI researchers working on interpretability interfaces, and MI researchers who want to understand how their tools might be used by non-experts. It deserves a serious referee—the framing is worth engaging with, even if the empirical support is thin. I'd send it to review but flag that the lay-user claim needs either tempering or a validation plan.\n\nBest,\n[Your name]","headline":"A well-structured conceptual paper that names a real gap—HCI lacks a paradigm for treating model internals as interaction material—but the central claim about lay users is undersupported and the framework's viability depends on MI reliability the paper itself flags as unresolved.","tokens_in":24704,"tokens_out":560,"duration_ms":87256},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"This paper argues that the inner representations of neural networks — what a model encodes between input and output — can serve as a new interaction material, letting users inspect, steer, and edit AI rather than only prompt it.","keywords":["Neural Representation Interaction","human-AI interaction","mechanistic interpretability","interaction paradigm","design space","representational affordances","steering vectors","sparse autoencoders"],"falsifier":"A field study in which lay users are given an inspect-steer-edit interface built from sparse-autoencoder features or verbalized activations of a real model and asked to produce a specified, targeted behavior change (say, suppressing a persona trait without altering unrelated outputs); if non-expert users cannot reliably achieve and verify such changes — or if causal tests with matched counterfactuals show the edited features do not actually drive the behavior — the paradigm's premise of readable, steerable representational bits fails.","tokens_in":25342,"feed_emoji":"🧠","tokens_out":9686,"duration_ms":60608,"temperature":0.7,"pith_summary":"Human-AI interaction today runs almost entirely through prompts and outputs, leaving the interior of the model a black box. This paper proposes Neural Representation Interaction (NRI): treating the internal representations of neural networks — the activations, features, circuits, and parameters that encode what the model has learned — as an interaction material that users can inspect, steer, and edit. The authors synthesize mechanistic interpretability into a four-dimensional design space (Interaction, Target, Referent, Purpose) and map existing interfaces onto it, finding that most serve expert auditors while few support lay users. If the paradigm holds, AI stops being an inscrutable oracle and becomes a system whose computation can be questioned, audited, and shaped directly; the paper sketches what that would look like in education, personal user modeling, and hiring audits.","feed_headline":"AI's inner representations become a material users can shape","feed_subtitle":"Defines a design space for inspecting, steering, and editing AI models — and finds lay users are left out.","key_machinery":"The central organizing device is the four-dimensional design space derived from General Morphological Analysis, whose facets are Interaction (Inspect, Steer, Edit), Target (Activations, Components, Circuits, Parameters), Referent (Concepts, Computation, User information, Behavior), and Purpose (Mechanistic Understanding, Sensemaking, Decision Support, Auditing, Control). It converts the scattered toolbox of mechanistic interpretability into a combinatorial grid in which each full interaction is a cell, letting the paper characterize existing systems and identify empty cells worth designing for. The material the space operates on is the set of 'representational bits' — the linear feature directions, sparse dictionary features, transformer components, and subgraph circuits that mechanistic interpretability extracts from model activations.","core_discovery":"The paper's central claim is that the semantic structure inside a trained neural network, not just the text it emits, is an interaction surface. It names the elementary units of that structure 'representational bits' and defines NRI as interaction that enables users to interpret and direct them, placed on the same footing as graphical, tangible, and bio-computer interaction. The case rests on three moves: the affordance logic that produced earlier interface paradigms applies to neural computation, since models encode concepts and algorithms that never appear in their outputs and those encodings can be read, pushed, and rewritten; the technical means already exist in mechanistic interpretability (linear probes, steering vectors, verbalization, sparse autoencoders, attribution, circuit discovery) and can be reframed as user-facing capabilities; and a design space built with General Morphological Analysis — Interaction (Inspect, Steer, Edit), Target (Activations, Components, Circuits, Parameters), Referent (Concepts, Computation, User information, Behavior), Purpose (Mechanistic Understanding, Sensemaking, Decision Support, Auditing, Control) — lets designers see which cells are occupied and which are open. Mapped onto this space, the tools built so far cluster around expert auditing, with TalkTuner, Neural Transparency, and Multi-Turn Neural Transparency as the rare lay-user exceptions. The authors state plainly that the design space presumes a transformer architecture and that the underlying methods are still works in progress.","pith_inferences":["The same mapping that shows most tools serve expert auditors suggests that the binding constraint on NRI is translational, not technical: the real open problem is re-presenting circuit tracers and feature dashboards as progressive, testable interfaces for lay users.","A testable consequence that follows from the paper's logic but is not tested there: interaction that lets users causally manipulate representations of themselves, as in Latent Mirror, should improve trust calibration and reduce over-reliance more than the descriptive explanations XAI dashboards provided — because the user is editing, not just reading.","The authors' own architectural caveat implies the design space is transformer-relative; a natural extension would be an architecture-relative Target dimension, where recurrent and state-space models map to compressed-state components and diffusion models map to dictionary features with no circuit analogue.","If personalized NRI interfaces succeed, mechanistic interpretability tools would gain a second life as consumer privacy surfaces: letting people see and edit what a model believes about them turns the paper's data-provenance concern into a direct user-facing feature rather than a policy footnote."],"forward_implications":["Human-AI interaction gains a fourth interaction paradigm alongside graphical, tangible, and bio-computer interfaces, with representational affordances — the coupling of a model's internal states with a user's ability to perceive and act on them — as its defining principle.","The mapping onto the design space shows that nearly all current interpretability interfaces occupy cells aimed at expert auditors and AI-safety teams, while only a handful (TalkTuner, Neural Transparency, Multi-Turn Neural Transparency) serve lay users; the underexplored cells mark where new NRI tools would sit.","Existing actionable mechanistic interpretability — probes, steering vectors, verbalization, sparse autoencoders, attribution and circuit discovery — becomes a design resource, so building NRI interfaces does not require new model science, only re-presentation of existing methods.","The speculative applications become concrete design targets: a teaching tool that decomposes literary style into sparse concepts, a transparent user model (Latent Mirror) that lets people inspect and edit what the AI believes about them, and a causal auditing tool for hiring that tests whether demographic proxies actually drive recommendations.","Two design principles follow directly: progressive disclosure (reveal model internals in layers of complexity) and iterative hypothesis testing (support users in forming and checking causal questions about the model's behavior)."],"supporting_citations":[{"why":"Supplies the working definition of mechanistic interpretability as reverse-engineering neural networks, the technical foundation the whole paradigm builds on.","marker":"[12]"},{"why":"Establishes the linear-representation and superposition hypotheses that make features legible and steerable; load-bearing for the Inspect and Steer modes.","marker":"[36]"},{"why":"Verbalization method that decodes activations into natural language; anchors the claim that internal states can be presented to non-expert users.","marker":"[41]"},{"why":"Training-free verbalization (Patchscopes-style); also cited in the paper's own limitation that verbal descriptions may confabulate what the model computes.","marker":"[44]"},{"why":"Introduces General Morphological Analysis, the methodology from which the four-dimensional design space is derived.","marker":"[99]"},{"why":"Representation engineering: linear directions and activation steering across models; the core evidence that representations can be read and directed.","marker":"[141]"},{"why":"TalkTuner, one of the three existing interfaces that serve lay users; load-bearing for the claim that few tools support everyday users.","marker":"[23]"},{"why":"Neural Transparency, another lay-user interface; part of the evidence for the expert-versus-lay asymmetry the paper reports.","marker":"[54]"},{"why":"Shows that not all language-model features are linear; the paper cites it as the key unresolved assumption behind steering methods.","marker":"[37]"}],"fun_headline_variants":[],"cache_read_input_tokens":0,"weakest_assumption_plain":"The whole paradigm rests on the assumption that what mechanistic interpretability surfaces — approximately linear feature directions, sparse-autoencoder decompositions, verbalized descriptions of activations — faithfully reflects the model's actual computation and can be read and acted on by people who are not interpretability experts; the paper itself concedes that feature linearity, autoencoder fidelity, and verbalization trustworthiness are all unresolved.","fun_headline_variants_meta":{"raw":{"variants":[]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":8e-05,"raw_usage":{"cost":7.98252e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":7.98252e-05,"upstream_inference_prompt_cost":7.94612e-05,"upstream_inference_completions_cost":3.64e-07},"total_tokens":1080,"prompt_tokens":1078,"completion_tokens":2,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":0}},"tokens_in":1078,"tokens_out":2,"cost_source":"openrouter_usage","duration_ms":1375,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":0,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T10:38:56.261879+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"A field study in which lay users are given an inspect-steer-edit interface built from sparse-autoencoder features or verbalized activations of a real model and asked to produce a specified, targeted behavior change (say, suppressing a persona trait without altering unrelated outputs); if non-expert users cannot reliably achieve and verify such changes — or if causal tests with matched counterfactuals show the edited features do not actually drive the behavior — the paradigm's premise of readable, steerable representational bits fails.","supporting_citations":[{"cited_title":"Mechanistic Interpretability for AI Safety -- A Review","cited_arxiv_id":"2404.14082","evidence_quote":"Supplies the working definition of mechanistic interpretability as reverse-engineering neural networks, the technical foundation the whole paradigm builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the linear-representation and superposition hypotheses that make features legible and steerable; load-bearing for the Inspect and Steer modes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Verbalization method that decodes activations into natural language; anchors the claim that internal states can be presented to non-expert users."},{"cited_title":"Bogdan, Emmanuel Ameisen, James Chen, Dzmitry Kishylau, Adam Pearce, Julius Tarng, Alex Wu, Jeff Wu, Yang Zhang, Daniel M","cited_arxiv_id":null,"evidence_quote":"Training-free verbalization (Patchscopes-style); also cited in the paper's own limitation that verbal descriptions may confabulate what the model computes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces General Morphological Analysis, the methodology from which the four-dimensional design space is derived."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Representation engineering: linear directions and activation steering across models; the core evidence that representations can be read and directed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TalkTuner, one of the three existing interfaces that serve lay users; load-bearing for the claim that few tools support everyday users."},{"cited_title":"Bridging the Gap in the Responsible AI Divides","cited_arxiv_id":"2603.14495","evidence_quote":"Neural Transparency, another lay-user interface; part of the evidence for the expert-versus-lay asymmetry the paper reports."},{"cited_title":"Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark","cited_arxiv_id":null,"evidence_quote":"Shows that not all language-model features are linear; the paper cites it as the key unresolved assumption behind steering methods."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}