Pith. sign in

REVIEW 4 major objections 4 minor 13 references

Breaking the Block: Preserving Data Continuity to Train Superior SAEs for Instruct Models

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes Finetuning-aligned Sequential Training (FAST), which processes each instruction instance separately instead of concatenating them into fixed-length blocks, and reports that it makes sparse autoencoders for instruct…

desk verdict Useful empirical result—sequential SAE training beats block training on instruct data—but the central causal claim is confounded by a 4x context-length difference, and the promised GSNR analysis is missing. read the letter →

arxiv 2506.07691 v2 pith:EV5DV3TK submitted 2025-06-09 cs.CL cs.LG

Jiaming Li , Haoran Ye , Yukun Chen , Xinyue Li , Lei Zhang , Hamid Alinejad-Rokny , Jimmy Chih-Hsien Peng , Min Yang This is my paper
classification cs.CLcs.LG
keywords sparseautoencodersmechanisticinterpretabilityinstruction-tunedmodelsblocktrainingsequentialfeaturespecialtokenssteering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the standard way of training sparse autoencoders (SAEs) on instruction-tuned language models—concatenating many chat instances into fixed-length blocks—introduces destructive gradient noise because attention can leak across semantically unrelated conversations. Its proposed alternative, FAST, feeds each conversation through the model as an independent sequence using the model's chat template, so the SAE sees activations that match the fine-tuning distribution. The paper reports that FAST sharply lowers reconstruction error, especially on the special tokens that delimit user and assistant turns, and that the SAEs it produces carry a much larger share of high-quality, interpretable features. If these results hold, SAE training for instruct models should follow the per-instance paradigm rather than the block-training habit inherited from pretraining.

What carries the argument

The central object is the training pipeline: FAST drops the concatenate-and-split step and feeds each chat-template-formatted conversation as a separate sequence through the instruct model, collecting activations in a producer-consumer mixing buffer that shuffles and refills half the buffer each round. What this preserves is the exact position of special tokens and the within-instance semantic continuity the model was fine-tuned on. The evaluation machinery consists of two comparisons: reconstruction MSE (including a special-token variant) and a GPT-4o-based feature-interpretability scoring pipeline, using both ReLU and JumpReLU SAE architectures.

What would settle it

Train BT(F) with each conversation padded to a block boundary with a separator so special tokens occupy the same positions as under FAST, using identical data and compute; if the reconstruction and interpretability gaps shrink to negligible size, semantic discontinuity is not the operative cause.

Watch

Extended reading notes

Core claim

The paper's central claim is that Finetuning-aligned Sequential Training (FAST) yields superior SAEs for instruct models by preserving each data instance's semantic continuity: instead of concatenating and re-splitting conversations into blocks, it feeds each chat-template-formatted instance into the model sequentially and trains the SAE on those activations. Concretely, on Qwen2.5-7B-Instruct FAST reports a log-scaled special-token MSE of 0.6468 versus 5.1985 and 1.5093 for block training on pretraining data (BT(P)) and on fine-tuning data (BT(F)); on Llama-3.2-3B-Instruct, 21.1% of FAST features receive the top interpretability scores versus 7.0% and 10.2%. The paper also reports that steering the SAE latents that activate most on special tokens improves generation quality within a specific amplification range. The authors attribute these gains to removing the attention leakage and semantic discontinuity caused by concatenating unrelated conversations during block training.

Load-bearing premise

The results rest on the premise that the improvements come from preserving semantic continuity rather than from the different placement of special tokens or other distribution shifts between FAST and block training, and the paper does not isolate this mechanism.

Editorial extensions

If this is right

  • SAE training for instruction-tuned models should use FAST's per-instance sequential processing instead of concatenating data into fixed-length blocks.
  • Special-token reconstruction error drops by a large margin under FAST, so SAEs intended to study instruction following and chat formatting should be trained this way.
  • Steering the SAE features that respond to special tokens moderately improves the model's answer quality, offering a new control handle for generation behavior.
  • Because FAST yields a higher share of top-scored interpretable features on every tested model, interpretability studies on chat models should adopt the sequential paradigm.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the discontinuity mechanism is the real driver, the same per-instance principle should also improve other activation-based training on chat data, such as training probes or linear classifiers, since concatenation would inject the same cross-example attention noise.
  • A padding-controlled comparison between FAST and BT(F) would separate the semantic-continuity effect from the simple fact that FAST places special tokens at their canonical positions; the paper does not report such an ablation.
  • The claimed GSNR analysis is absent from the body, so the mechanism remains a hypothesis; measuring gradient signal-to-noise during the two training regimes would be a direct test.
  • The sweet-spot steering behavior suggests special-token features act as soft instruction-following controls; a natural extension is to search for similar features for refusal or safety tokens in larger instruct models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Finetuning-aligned Sequential Training (FAST), a method for training sparse autoencoders (SAEs) on instruction-tuned LLMs. Instead of concatenating dataset instances into fixed-length blocks as in Block Training (BT), FAST feeds each data instance independently, using the model's chat template, with a truncation length of 8,192 tokens. The authors compare FAST against BT over pretraining data (BT(P)) and over the same finetuning data (BT(F)) across seven Llama and Qwen instruct models, two SAE families (Standard and JumpReLU), and multiple layers. They report that FAST yields lower MSE, especially for special-token reconstruction, and higher GPT-4o-rated feature interpretability, and they demonstrate special-token steering effects on generation quality. The paper claims a GSNR-based theoretical characterization of destructive gradient noise from block concatenation, though no GSNR analysis appears in the manuscript.

Significance. If the central comparison were clean, the result would be practically significant: SAE training on instruct models is an active area, and a simple, open-sourced training-paradigm change that improves both reconstruction and interpretability would be a useful contribution. The experimental breadth is a clear strength: the paper covers 7 models, multiple layers, two SAE variants, and provides full MSE tables in Appendix D, with training logs in Appendix G. The use of sae_lens as a third-party baseline implementation and the same finetuning dataset for BT(F) and FAST are also strengths that reduce concerns about self-authored baselines. However, the headline comparison is confounded by a mismatch in context length, and the claimed theoretical mechanism (GSNR) is absent from the body; these issues must be addressed before the empirical claims can be taken as established.

major comments (4)
  1. [§4.1, Configuration; §3, Motivation] The FAST versus BT(F) comparison is confounded by context length. Section 4.1 specifies that for BT(P) and BT(F) context_size is 2,048, while FAST has no explicit context_size and instead uses a truncation length of 8,192. The two conditions therefore differ simultaneously in (i) whether instance boundaries are preserved and (ii) the attention context available at each token. Since SAEs are trained on hidden activations, which depend on the preceding context, the reported MSE and interpretability gains (e.g., Section 4.2's 0.6468 vs 1.5093) may be driven by the 4x longer context rather than by sequential processing. The paper attributes the gains to removing semantic discontinuity and attention leakage (Abstract, Section 3), but no experiment isolates that factor. A control with matched context length—e.g., BT(F) with context_size 8,192, or FAST with truncation 2,048—is needed to substantiate the central causal claim.
  2. [Abstract; §3] The abstract and Section 3 state that the paper uses GSNR analysis to "theoretically characterize" the destructive gradient noise from block concatenation, and the abstract reports "higher GSNR" for FAST. However, no GSNR definition, derivation, or experimental result appears anywhere in the main text or the appendices. As written, the theoretical mechanism is unsupported, and the GSNR claim in the abstract is unverifiable. The authors should either add a concrete GSNR analysis (definition, computation on the relevant activations, and a comparison across conditions) or remove the GSNR claims from the abstract and motivation.
  3. [§4.2; Eq. (7); Table 6] The headline numbers are reported ambiguously. In §4.2, the text says "In Qwen2.5-7B-Instruct-L18-8X-Standard, FAST achieves 0.6468, outperforming the baselines (5.1985 and 1.5093)" and elsewhere calls this an MSE. However, Eq. (7) defines raw MSE, while the reported values are log2-transformed MSEst values from Table 6. The footnote mentions the log2 transform, but the abstract and Section 1 describe these as "mean squared error" or "log-scaled MSE" without consistently stating that the headline is log2(MSE) for special tokens only. This conflation makes the magnitude of the improvement difficult to interpret and should be corrected with explicit notation throughout, e.g., "log2(MSEst)."
  4. [§5, Feature Interpretability] The interpretability comparison is based on GPT-4o scores of only 128 randomly sampled features per SAE after filtering dead features. The text does not report the number of seeds, the variance of the percentages, or the number of dead features per method. Since the headline claim (21.1% vs 7.0% and 10.2% for Llama3.2-3B-Instruct) depends on a single 128-feature sample, the result may not be stable. The authors should provide confidence intervals, multiple random draws, or a larger evaluation sample, and should also report the dead-feature fractions, because differential dead-feature rates across BT(P), BT(F), and FAST could bias the comparison.
minor comments (4)
  1. [Throughout] There are frequent spacing artifacts in the paper's rendering, such as "F AST" instead of "FAST" (Abstract, Section 1), and "metioned" in Section 4.1. These should be fixed in a final revision.
  2. [Table 18] The model name is misspelled as "Qwen2.5-0.5B-insturct" in the table caption; it should be "Qwen2.5-0.5B-Instruct."
  3. [§4.2; Figure 4 and 6] The figures show log2(MSE) on the y-axis but the axis label says "MSE Loss-log2-Special Token." Consider adding "log2" to the axis title or a caption note to avoid confusion.
  4. [Appendix C] The hyperparameter table lists "L sparsity" when the text in Section 3.2 uses lambda for the sparsity coefficient; use consistent notation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: FAST's headline gains are measured against independently trained block-training baselines, and no load-bearing conclusion reduces by construction or self-citation to its own inputs.

full rationale

The paper's claimed derivation chain is that semantic discontinuity from concatenating chat instances introduces destructive gradient noise, and that per-instance sequential training removes it, improving SAE reconstruction and interpretability. The headline numbers (MSE 0.6468 vs 5.1985/1.5093; 21.1% vs 7.0%/10.2%) are empirical comparisons between independently trained SAEs: BT(F) uses the same finetuning mixture as FAST, both are implemented through the third-party sae_lens library, and the motivating discontinuity claim is attributed to external work (Kissane et al.). No equation defines FAST's advantage via the metric used to report it, no fitted parameter is renamed as a prediction, and no conclusion rests on a self-authored citation. The abstract promises a GSNR analysis that does not appear in the body, and Section 4.1 sets context_size 2,048 for BT(P)/BT(F) but an 8,192 truncation length for FAST, so the comparison is not fully controlled; these are support and experimental-design gaps rather than circular reductions. The evaluation dialogues are drawn from the same instruction-data distribution FAST is designed to match, which is a distribution-match property of the method, not a derivation equivalent to its own inputs. Verdict: no significant circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard SAE machinery borrowed from cited work, plus several domain assumptions about how interpretability is measured. No new entities are introduced. The most delicate load-bearing assumption is the asserted causal mechanism for block-training degradation, which the abstract says is characterized by GSNR but the body never derives; if that mechanism is wrong, the sequential-versus-block framing is an overstatement, though the empirical comparison would still stand. The steering analysis additionally requires selecting alpha per question post hoc.

free parameters (6)
  • Sparsity regularization coefficient lambda (Standard SAE) = 5
    Chosen per architecture in Appendix C; reconstruction quality and feature sparsity both depend on it, and no sensitivity analysis over lambda is reported.
  • Sparsity regularization coefficient lambda (JumpReLU SAE) = 0.01
    L0-target style coefficient from Appendix C; the reported MSE and interpretability comparisons all use this single value.
  • JumpReLU bandwidth and init threshold = 0.001, 0.001
    Appendix C hyperparameters for the JumpReLU activation; results could shift with these tuning choices.
  • Steering coefficient alpha = 15 to 200 per question
    In Section 6, the claimed sweet spot for quality improvement is selected per question after observing outputs; the beneficial range is identified retrospectively, and the paper's Limitations call the steering work preliminary.
  • N-gram size for deduplication = 20
    Appendix B: the 20-gram dedup threshold that reduces 11.4M samples to 4.76M is chosen by hand and shapes both training and evaluation distributions.
  • Training token budget = 40,960,000
    Appendix C: all methods are matched at this token count; FAST uses up to 8,192-token truncation while baselines use 2,048-token blocks, so token counts are equal but sequence structure differs.
assumptions (5)
  • domain assumption Residual stream activations are a sufficient substrate for interpretable SAE features
    Section 4.1 follows prior works (Bricken et al., Gao et al.) with no in-paper justification.
  • domain assumption GPT-4o rubric scoring on five sentences per feature is a valid measure of feature interpretability
    Section 5 relies on this to produce the 21.1% versus 7.0% and 10.2% headline; no inter-annotator or model-stability analysis is given.
  • ad hoc to paper Semantic discontinuity from block concatenation, not data distribution, causes the observed degradation in BT(F)
    Section 3 Motivation asserts this mechanism, and the abstract promises a GSNR characterization, but no GSNR analysis appears in the body; the causal claim is unmeasured.
  • domain assumption Middle layers are the best layers for interpretability evaluation
    Section 5 selects middle layers based on demonstrated superior performance without showing the supporting experiment.
  • domain assumption JumpReLU features can be evaluated on strongly activated sentences only; dead and weakly activated features can be ignored
    Section 5 filters features with no activation in their top-5 sentences; the paper lists weak-activation coverage as a limitation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Breaking the Block: Preserving Data Continuity to Train Superior SAEs for Instruct Models." pith.science (2026). https://pith.science/paper/EV5DV3TK

@misc{pith2026250607691,
  author       = {Pith},
  title        = {Pith review of: Breaking the Block: Preserving Data Continuity to Train Superior SAEs for Instruct Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EV5DV3TK}},
  note         = {Machine review of arXiv:2506.07691}
}
read the original abstract

Sparse Autoencoders (SAEs) are a cornerstone of mechanistic interpretability. Existing training methods inherit the Block Training paradigm from LLM pre-training, which introduces destructive gradient noise in instruct models due to attention leakage from unrelated contexts. Using GSNR analysis, we theoretically characterize this issue and propose Finetuning-aligned Sequential Training (FAST), a sequential training paradigm specifically designed for instruct models. FAST aligns SAE training with the data distribution and activation patterns of instruct models, substantially improving both reconstruction fidelity and feature interpretability. Experimental results show that FAST achieves higher GSNR, a significantly lower log-scaled MSE of 0.6468 compared to the baseline's 5.1985, and a near-zero Delta Loss (-0.51\% to 0.37\%). Moreover, on Llama-3.2-3B-it, FAST produces 21.1\% high-quality features, substantially outperforming baseline methods that achieve 7.0\% and 10.2\%. We further find that intervening on special token activations through SAEs can improve generation quality, revealing new opportunities for fine-grained control. Our codes are available as open source at https://github.com/Geaming2002/FAST.

Figures

Figures reproduced from arXiv: 2506.07691 by the authors.

Figure 1
Figure 1. Overview of the sparse autoencoder, illustrat [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the LLM training pipeline and SAE training methods. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The mixing activation buffer is shuffled, half [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: MSEst performance of the JumpReLU SAE (all metrics are presented in log scale, where lower values indicate better SAE reconstruction performance). Within the JumpReLU architecture, FAST exhibits the best reconstruction capability compared to BT(P) and BT(F). Model Name…
Figure 5
Figure 5. Figure 5: Experiment results of feature interpretability. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: MSEst performance of the Standard SAE (all metrics are presented in log scale, where lower values indicate better SAE reconstruction performance). Within the Standard architecture, FAST exhibits the best recon￾struction capability compared to BT(P) and BT(F) While the …
Figure 7
Figure 7. Figure 7: The steering output generated by Qwen2.5-7B-Instruct with Feature ID: 13794, focusing on [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: The steering output generated by Llama3.1-8B-Instruct with Feature ID: 22642, focusing on [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]
Figure 9
Figure 9. Figure 9: The steering output generated by Qwen2.5-7B-Instruct with Feature ID: 13794, focusing on [PITH_FULL_IMAGE:figures/full_fig_p028_9.png]
Figure 10
Figure 10. Figure 10: The steering output generated by Llama3.1-8B-Instruct with Feature ID: 22642, focusing on [PITH_FULL_IMAGE:figures/full_fig_p029_10.png]
Figure 11
Figure 11. Figure 11: The steering output generated by Qwen2.5-7B-Instruct with Feature ID: 13794, focusing on [PITH_FULL_IMAGE:figures/full_fig_p030_11.png]
Figure 12
Figure 12. Figure 12: The steering output generated by Llama3.1-8B-Instruct with Feature ID: 22642, focusing on [PITH_FULL_IMAGE:figures/full_fig_p030_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 9 canonical work pages

  1. [1]

    Analyze the context provided, which consists of a sequence of alternating tokens and their cor- responding activation values

    work page

  2. [2]

    Assign a score based on the activation consistency rubric

    work page

  3. [3]

    s race, Olympic 5,000m champion Vivian Cheruiyot of Kenya won in her first half marathon.\nScotland\

    Provide a descriptive name for the feature that captures its essence. Example output: ’My final verdict score is: [[3]], feature name is [[Mathematical Problem Explana- tion]]’. User: {prompt} Prompt Template Below is the context of feature {feature_index}, represented as sentences with tokens and their activation values: {context} F Implementation Detail...

    work page 2021

  4. [4]

    Adly Templeton

    The ai risk repository: A comprehensive meta- review, database, and taxonomy of risks from artifi- cial intelligence.arXiv preprint arXiv:2408.12622. Adly Templeton. 2024.Scaling monosemanticity: Ex- tracting interpretable features from claude 3 sonnet. Anthropic. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zh...

    arXiv 2024

  5. [8]

    Quel est le poste pour lequel vous postulez ?

    work page

  6. [9]

    Quelle est l'entreprise à laquelle vous postulez ?

    work page

  7. [10]

    Quelles sont vos principales compétences et expériences pertinentes ?

    work page

  8. [11]

    <|im_start|>

    Quels sont vos objectifs professionnels à court et long terme ? Avec ces informations, je pourrai vous proposer un modèle de lettre de motivation personnalisé et adaptée à votre situation.<|im_end|> Steered Output ()α=15 V oici un exemple de lettre de motivation pour un poste d'ingénieur en informatique : [V otre nom et adresse] [Date] [Nom et adresse de ...

    work page

Show all 13 references
  1. [12]

    功能：铁锤主要⽤于物理操作，如敲击、拆卸或组装物体；⽽磁铁主要⽤于吸引或排斥具有磁性的物体。

  2. [13]

    <|start_header_id|>

    特性：铁锤是⼀个物理⼯具，没有磁性；磁铁则具有吸引特定⾦属的能⼒，这是由于其内部的磁场。 综上所述，铁锤和磁铁之间的主要区别在于它们的功能和特性。<|im_end|> Figure 11: The steering output generated by Qwen2.5-7B-Instruct with Feature ID: 13794, focusing on user and <|im_start|>tokens for the Question 4 (entity discrimination task). Llama3.1-8B-Inst...

    2022

  3. [2013]

    InProceedings of the 2013 conference of the north american chapter of the as- sociation for computational linguistics: Human lan- guage technologies, pages 746–751

    Linguistic regularities in continuous space word representations. InProceedings of the 2013 conference of the north american chapter of the as- sociation for computational linguistics: Human lan- guage technologies, pages 746–751. Arindam Mitra, Luciano Del Corro, Guoqing Zhen...

    2013 arXiv

  4. [2022]

    Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda

    Training language models to follow instruc- tions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. 2024a. Improving...

    arXiv

  5. [2024]

    https://github.com/jbloomAus/ SAELens

    Saelens. https://github.com/jbloomAus/ SAELens. Connor Kissane, Robert Krzyzanowski, Arthur Conmy, and Neel Nanda. 2024a. Saes (usually) transfer be- tween base and chat models. Alignment Forum. Connor Kissane, Robert Krzyzanowski, Neel Nanda, and Arthur Conmy. 2024b. Saes are...

    2003 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.