{"id":"a4cbbd50-ca7e-47d3-978d-047d11ca630d","arxiv_id":"2607.25132","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"In a two-rule GOHR task, SAEs on frozen Transformer decision-token embeddings recover selective shape/bucket concepts and interpretable probe-then-switch strategy dimensions.","lead":"Sparse autoencoders recover shape, bucket, and probe-and-switch strategy features inside a small Transformer agent that must infer one of two hidden Game-of-Hidden-Rules mappings from accept/reject feedback alone. The work is a controlled lab-scale test of whether post-hoc SAEs expose behaviorally meaningful structure in sequential decision agents.","discovery_kind":"new_application","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The headline \"concept recovery\" metrics are partially guaranteed by construction: the evaluated concepts (chosen shape, chosen bucket) are components of the very action logits the SAE is trained to reconstruct (α=0.1 head, §2.3), and the OR-combined recall can saturate regardless.","rationale":"The reader identified the same load-bearing soft spot — the auxiliary action head plus OR-combined recall — and priced it correctly into a CONDITIONAL verdict with medium correctness risk. My pass sharpens it in two ways but does not change its direction. First, the circularity is stronger than \"partial\": for the bucket concept specifically, the concept label is a component of the action whose logits the SAE objective explicitly reconstructs, and the reported numbers (bucket nAUC = 1.000 alongside OR-group precision 0.513) display precisely the recall-by-redundancy signature the paper's own Scope section warns about. Second, the missing baselines (α=0 ablation, linear probe, random dictionary, group-size cap) are what make this unresolvable from the text alone — a standard interpretability-paper control set, all feasible with the authors' existing saved activations. I keep the verdict at CONDITIONAL rather than pushing to REJECT because: the paper is appropriately scoped (it explicitly restricts its strongest conclusion to the two-rule setting in §3.6 and proposes per-dimension statistics as next steps); the tier-(b) qualitative evidence (Fig. 3 heatmaps tracking the probe-then-switch action change, Fig. 4 dimensions with sensible semantics) is directionally consistent and not itself circular; and the proposed controls are cheap and would plausibly rescue much of the claim. The reader's conditions (per-dimension stats, ablation without the action head, artifact release) are exactly the right ones; I would add the linear-probe and group-size-cap controls to the list.","tokens_in":8187,"tokens_out":1867,"duration_ms":66043,"concrete_test":"Retrain the SAE with α=0 (no action-reconstruction head) on the same saved embeddings, and additionally fit multinomial linear probes from the raw decision-token embeddings to the shape and bucket labels. Recompute the recall-vs-precision-threshold curves (Fig. 2 / Table 4) for both controls, with the OR-group size capped at the same count used in the main run. Interpretation: if α=0 SAE and the linear probe both reproduce bucket nAUC ≈ 1.0, the tier-(a) result reflects trivial decodability, not SAE-specific recovery; if bucket recall collapses without the action head, the reported recovery is an artifact of the objective. Either outcome materially weakens the headline claim; survival of the curves under both controls would meaningfully strengthen it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two tiers: (a) aggregate recovery of shape/bucket concepts (Table 4, Fig. 2) and (b) discovery of rule-conditioned strategy dimensions (Fig. 4). Tier (a) is the quantitative backbone, and it is the least secure. The action-reconstruction term α‖G(z)−ℓ‖² trains z to linearly predict the policy logits, and the policy's output at ⟨bucket_cls⟩ *is* the bucket decision — so \"SAE dimensions selective for the chosen bucket\" is close to restating the auxiliary objective, not an emergent finding. The numbers are consistent with this: bucket concepts hit nAUC = 1.000 / recall = 1.000, which looks suspiciously perfect, while the OR-group precision at τ=0.8 is only 0.513 — meaning nearly half the decisions where \"some bucket-selective feature fires\" do not involve that bucket. Perfect recall plus mediocre group precision is exactly the signature of recall-by-redundancy under the logical-OR metric (a failure mode the paper itself flags in §3.6): with enough features in D_c(τ), P(∃j: z_j>0 | c=1) trivially approaches 1. No group-size budget or random-dictionary control is reported, so we cannot tell whether nAUC would also be ~1 for a random sparse code of the same width. There is also no linear-probe baseline: shape and bucket are literal input tokens, so a linear probe on the raw embeddings plausibly matches the SAE curves, which would demote tier (a) from \"SAE recovers concepts\" to \"concepts are linearly decodable,\" a much weaker and well-known statement. Tier (b) — the genuinely interesting claim about probe-and-switch dimensions — rests on two hand-picked dimensions (9697, 8720) with 64-example profiles and no per-dimension selectivity statistics, seeds, or error bars. If tier (a) is largely an artifact, the paper's quantitative support collapses onto these two qualitative examples.","agreement_with_reader":"agree"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"This technical report trains a small tokenized autoregressive Transformer (6 layers, d=256) via SFT warm start plus A2C on a two-rule instance of the Game of Hidden Rules, where both hidden rules are shape-to-bucket permutations (Table 1) chosen so that every shape maps to a different bucket across rules. The frozen policy is shown to exhibit a probe-and-switch strategy in rollouts (Fig. 3). The authors then train sparse autoencoders with an auxiliary action-reconstruction head (Eq. for L_SAE, α=0.1) on saved decision-token embeddings and report two results: (a) aggregate recovery of shape and bucket concepts, measured by recall of OR-combined feature groups D_c(τ) as a function of per-dimension precision threshold τ (Fig. 2, Table 4), and (b) individual SAE dimensions with interpretable rule-conditioned semantics, e.g., a dimension selective for circle→bucket-1 probe moves that succeed ~half the time (Fig. 4). The paper is explicit about scope (§3.6): two closely related rules, qualitative rollout evidence, and a feature-group metric that can hide redundancy.","tokens_in":8619,"tokens_out":3022,"duration_ms":100981,"significance":"If the results hold under appropriate controls, the paper provides a clean, fully checkable case study of SAE interpretability in a sequential decision-making agent: the latent variable (rule identity) is known to the experimenter, the expected strategy is simple, and recovered features can be cross-validated against rollout behavior. The three-view methodology (rollouts, concept-level recall curves, per-dimension profiles) is a reasonable template, and the two-rule probe-and-switch diagnostic is well designed for falsifiability — the pre-feedback unidentifiability argument is correctly stated. The report is commendably honest about its limitations (§3.6) and does not overclaim beyond the two-rule setting. However, the quantitative backbone (Table 4) currently lacks the baselines and controls needed to distinguish \"SAE recovers concepts\" from \"concepts are linearly decodable from action-predictive codes,\" and the per-dimension findings rest on two hand-picked dimensions from a single run. As written, this is a useful but preliminary case study rather than a validated method demonstration.","major_comments":[{"comment":"Circularity risk in the aggregate-recovery claim. The SAE objective includes α‖G_η(z)−ℓ‖² with α=0.1, training z to linearly predict the policy's action logits. The evaluated concepts are components of that very action: the bucket decision is read out at ⟨bucket_cls⟩ (the policy factorization in §2.2 makes k_t an explicit output), and the chosen shape is determined by the chosen coordinates, also policy outputs. Table 4's bucket row — nAUC = 1.000, Recall@0.8 = 1.000 — is therefore close to a restatement of the auxiliary objective rather than an emergent finding, and the perfect score is more consistent with reconstruction-by-construction than with selective recovery. The load-bearing control is an ablation at α=0 (and ideally a sweep over α): does bucket-concept recall survive when the SAE is trained on reconstruction + sparsity alone? Without it, tier (a) of the central claim is not es","section":"§2.3, §3.3, Table 4"},{"comment":"The OR-combined recall metric is not controlled for feature-group size, and the reported numbers show the signature the paper itself warns about in §3.6. Recall(c,τ) = P(∃j∈D_c(τ): z_j>0 | c=1) increases mechanically with |D_c(τ)|, yet Table 4 reports only the number of concept groups (n=4/8), never the number of selected dimensions per concept at each τ. The bucket group at τ=0.8 has recall 1.000 but group precision 0.513 — nearly half the decisions where some bucket-selective feature fires do not involve that bucket — which is exactly the recall-by-redundancy pattern. There is also concrete evidence of group overlap: in Fig. 3(a), t=0, the Triangle and Bucket 2 cells both read 3.92, implying the same dimension attains the max in two different 'concept-specific' groups; the same pattern repeats (1.92 at t=1 across Triangle/Bucket 3; 4.56 at t=0 in panel (b) across Star/Bucket 3). Please","section":"§2.3, Table 4, Fig. 2"},{"comment":"Possible selection leakage in the precision/recall estimates. The caption states the curves are computed over 10,000 held-out activation samples, but it is not stated whether Prec(j,c) — used to select D_c(τ) — is estimated on a split disjoint from the samples on which Recall(c,τ) is evaluated. If the same 10,000 samples are used for both selection and evaluation, the recall curves are optimistically biased by selection, and the bias grows with the dictionary size. This is a cheap fix (split the held-out set) but it is load-bearing for every number in Table 4.","section":"§3.3, Fig. 2 caption"},{"comment":"Statistical robustness is insufficient for both tiers of the claim. The SAE analysis appears to be a single run ('the run analyzed here', §2.3) on a single trained policy, with no seeds, no error bars on Fig. 2, and no statement of dictionary width or sparsity level (dimension index 9697 implies a wide dictionary, which matters for interpreting per-dimension selectivity as a multiple-comparison problem: with ~10⁴ dimensions, some will look clean by chance, especially when profiles are summarized by the 64 highest-activation examples). At minimum: report dictionary size, β, activation frequency statistics, and the selection procedure that surfaced dims 9697 and 8720; ideally repeat policy training and SAE fitting over a few seeds and report variance of nAUC. Similarly, the behavioral claim of probe-and-switch rests on two displayed episodes (Fig. 3); a quantitative rollout statistic (e.g.","section":"§3.5, Fig. 4; §3.2"}],"minor_comments":[{"comment":"The choices α=0.1 and β (unreported) are never justified or ablated; the sparsity penalty value and resulting average L0 should be stated in Table 2 or a companion table.","section":"§2.3"},{"comment":"SFT exact accuracy of 0.436 and bucket accuracy of 0.576 are quite low; a sentence clarifying that this is expected given multiple valid targets (and that A2C is where rule inference is learned) would help readers not misread the warm start as the learned policy's quality.","section":"§3.1, Table 3"},{"comment":"Notation: E_ϕ, D_ψ, G_η are introduced but the architectures (hidden width of z, whether G is linear as stated 'linear action-reconstruction head') are not specified; 'linear' is asserted in text but G_η suggests parameters — please reconcile.","section":"§2.3"},{"comment":"The caption's explanation for asymmetric shape curves (policy may prefer clearing one shape first) is plausible but unchecked; a one-line statistic on per-shape decision frequencies would substantiate it.","section":"Fig. 2"},{"comment":"The SAE literature cited is minimal ([1] only); recent work on SAE evaluation methodology and on the difficulty of distinguishing interpretable-from-decodable features (e.g., downstream SAE critiques and baseline comparisons in LM interpretability) should be cited and engaged, since it bears directly on the baseline concerns above.","section":"References"},{"comment":"The phrase 'recover this structure' is stronger than what the OR-recall metric establishes; 'are predictive of' or 'align with' would better match the evidence as currently controlled.","section":"Abstract/Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a modest, honestly-scoped technical report and the authors clearly know the weaknesses — §3.6 concedes most of them. The problem is that the concessions do not come with the corresponding controls: the α-head ablation, linear-probe and random-dictionary baselines, disjoint selection/evaluation splits, and seed statistics are all inexpensive in this small setting and would either substantially strengthen or substantially qualify the headline result. I lean toward thinking the qualitative per-dimension findings (Fig. 4) will survive and the aggregate Table 4 numbers are the vulnerable part. Given the DARPA-funded, report-style nature of the venue, major revision rather than rejection seems right; if the venue cannot accommodate a revision cycle, this is borderline."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a small, readable case study showing that a frozen tokenized Transformer on a two-rule GOHR task does probe-and-switch, and that post-hoc SAEs on decision tokens light up shape/bucket and a couple of strategy-like dimensions. It does not invent a new method; it packages a controlled sandbox where you can check SAE features against known latent structure.\n\nWhat is actually new is the combination: GOHR’s two opposing shape-to-bucket maps, a history-conditioned policy with no rule label, frozen rollouts, then SAEs on ⟨x/y/bucket_cls⟩ states with concept-recall curves and a few hand-interpreted dims (circle→bucket-1 probes; successful square→bucket-2). The write-up is clear, the figures line up with the story, and the Scope section owns the main limits—two related rules, qualitative rollouts, OR-combined groups hiding redundancy. Citations are appropriate (Cunningham SAE, A2C, Olah, prior GOHR). Math is standard dictionary learning plus an auxiliary logit head; nothing broken.\n\nSoft spots, in proportion. The stress-test lands on tier (a): α=0.1 trains z to reconstruct action logits, and “chosen bucket” is literally what the policy outputs at bucket_cls, so perfect bucket nAUC/recall is close to restating the objective. Group precision ~0.5 at τ=0.8 plus OR-recall is exactly the redundancy signature they flag themselves. No linear-probe baseline on raw h, no random-dictionary control, no seeds/error bars, no code. Tier (b)—probe-and-switch dims—is the more interesting claim and rests on two example dimensions with top-64 profiles. That is fine for a report if you do not oversell “concept recovery.” Circularity is partial, not fatal; behavioral rollouts still stand on their own.\n\nWho it is for: people building interpretability testbeds for sequential agents, or anyone who wants a concrete GOHR+SAE recipe before scaling rule mixtures. Not for someone hunting a general theory of concepts in RL.\n\nI would send it to peer review as a short empirical/methods note, with referees asked for ablations (no action head; probe baseline; per-dim stats) and artifact release. Engage if you care about small rule-learning sandboxes; skip if you need a methodological leap.","headline":"Clean two-rule SAE case study with honest scope; the quantitative backbone is partly baked in by the action head and OR-recall, so the real interest is the strategy dimensions and the testbed framing.","tokens_in":9818,"tokens_out":611,"would_cite":false,"duration_ms":17601,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Sparse autoencoders recover shape, bucket, and probe-and-switch strategies inside a frozen two-rule GOHR agent.","keywords":["sparse autoencoders","interpretability","Game of Hidden Rules","reinforcement learning","Transformer policy","concept recovery","probe-and-switch","sequential decision-making"],"falsifier":"Retrain or ablate the SAE without the action-reconstruction term and recompute the recall-vs-precision curves and top-dimension behavior profiles: if shape/bucket coverage and probe-and-switch dimensions collapse, the reported concept recovery is largely an artifact of that auxiliary objective and the OR metric.","tokens_in":9487,"feed_emoji":"🔍","tokens_out":892,"duration_ms":18701,"temperature":0.7,"pith_summary":"This paper asks whether a small Transformer policy that must infer a hidden rule from accept/reject feedback actually stores rule-relevant concepts in its internal states. In a two-rule Game of Hidden Rules task, both rules map shapes to buckets with different permutations, the agent never sees a rule label, and the correct mapping is only identifiable after an informative probe. After supervised warm-start and A2C training, the frozen policy behaves like a probe-and-switch strategy: try one mapping, then switch after rejection. Sparse autoencoders trained on decision-token embeddings recover that structure. Concept-selective SAE dimensions cover most held-out decisions labeled by chosen shape or bucket, and individual dimensions line up with concrete strategies such as circle-to-bucket-1 probes and successful square-to-bucket-2 moves. A sympathetic reader cares because this is a controlled sequential setting where latent structure is known, behavior can be checked in rollouts, and post hoc features can be tied to both aggregate concepts and rule-conditioned actions.","feed_headline":"SAEs find probe-and-switch concepts in a two-rule agent","feed_subtitle":"Frozen GOHR policy states yield shape, bucket, and strategy features without ever seeing a rule label","key_machinery":"Post hoc sparse autoencoders on decision-token hidden states, trained with reconstruction, sparsity, and an auxiliary action-logit reconstruction head; concepts are scored by per-dimension precision and feature-set recall of the OR-combined selective dimensions.","core_discovery":"In a compact two-rule GOHR setting, sparse autoencoders trained on a frozen tokenized Transformer’s decision-token embeddings recover both aggregate task concepts (selected shape and bucket) with high feature-set recall at high per-dimension precision, and individual dimensions whose activations match interpretable rule-conditioned strategies such as probing one hypothesis and switching after negative feedback.","pith_inferences":["If the same SAE readout works when more than two permutations or attribute-irrelevant distractors are added, GOHR becomes a scalable ladder for testing whether dictionary learning tracks latent rule inference rather than surface board tokens.","Comparing SAEs with and without the action head would separate ‘features that explain actions’ from ‘features that merely reconstruct hidden states.’","The qualitative episode heat maps suggest a practical audit loop: flag decisions where strategy-linked dimensions fire without matching accept/reject history."],"forward_implications":["Small rule-learning environments with known latent structure can serve as testbeds for SAE interpretability before larger agents or richer rule mixtures.","Probe-and-switch behavior in frozen rollouts can be linked to specific SAE dimensions rather than only to external action traces.","Concept-level feature-set recall can be checked against simple labels such as chosen shape and bucket even when the policy never receives a rule label.","Alignment-oriented concept exposure can begin with sequential decision agents whose hidden variables are experimenter-known and behaviorally verifiable."],"fun_headline_variants":["SAEs recover probe-and-switch strategies in a two-rule GOHR agent","Sparse autoencoders extract shape and bucket features from frozen GOHR states","Decision-token SAEs reveal rule hypotheses without any rule labels","SAEs find selective strategy dimensions in a frozen two-rule policy","GOHR agent embeddings yield SAE features for probing and switching"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The claim depends on treating high OR-combined feature-group recall, under an SAE objective that also pushes the code to reconstruct actions, as genuine recovery of internal concepts rather than redundant behavior-aligned dimensions.","fun_headline_variants_meta":{"raw":{"variants":["SAEs recover probe-and-switch strategies in a two-rule GOHR agent","Sparse autoencoders extract shape and bucket features from frozen GOHR states","Decision-token SAEs reveal rule hypotheses without any rule labels","SAEs find selective strategy dimensions in a frozen two-rule policy","GOHR agent embeddings yield SAE features for probing and switching"]},"model":"grok-4.5","effort":"low","cost_usd":0.004718,"raw_usage":{"total_tokens":1320,"prompt_tokens":746,"num_sources_used":0,"completion_tokens":98,"cost_in_usd_ticks":47184000,"prompt_tokens_details":{"text_tokens":746,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":476,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":746,"tokens_out":98,"duration_ms":9790,"temperature":1.0,"reasoning_tokens":476,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T00:27:35.449835+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Retrain or ablate the SAE without the action-reconstruction term and recompute the recall-vs-precision curves and top-dimension behavior profiles: if shape/bucket coverage and probe-and-switch dimensions collapse, the reported concept recovery is largely an artifact of that auxiliary objective and the OR metric.","supporting_citations":[],"review_version":1}