{"id":"a1f7b0f5-50a9-41ff-a7c7-904cbb4369c6","arxiv_id":"2506.02204","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A pipeline that combines contextual embeddings with two LMs' per-word probabilities and sparse autoencoders to automatically find interpretable slices where one model outperforms another.","lead":"BehaviorBox automatically discovers fine-grained, readable features of text where one language model assigns consistently higher probabilities than another, such as \"conditional were after if you\" or \"tabs in narrative text\". It uses sparse autoencoders on word embeddings augmented with the two models' probabilities, giving practitioners a new way to see which specific contexts drive model differences.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run SAE instability is the load-bearing risk: all insights and generation hypotheses come from one seed per comparison, and Appendix A.1 concedes feature-space sensitivity to seeds; multi-seed evidence is needed before the claimed generalizations can be trusted.","rationale":"The paper's central claim has two parts: BehaviorBox discovers coherent features of performance differences, and those features generalize beyond the analysis corpus. I examined the support for each. The qualitative sections are carefully illustrated and the authors are appropriately cautious in the Limitations, but the entire evidentiary chain rests on features from a single SAE run per comparison. The reader's weakest assumption identifies exactly this, and I agree. Appendix A.1 is decisive: the authors themselves state that different seeds or data orderings \"can largely impact the space of features found by SAEs\" and report only informal observations of partial overlap. Without quantifying overlap across runs, Section 7's insights cannot be distinguished from run-specific artifacts. Section 8's generation experiment is the strongest evidence in the paper, but its hypotheses are selected post hoc from the same single run; the absence of a pre-registered or exhaustive feature-to-string mapping, and the lack of multiple-comparison correction, means the 6/8 significant results are upper-bound evidence rather than confirmation of stability. A multi-seed replication is therefore the decisive test: it would either show the core features and directions are robust (supporting the claim) or reveal that the reported insights are seed-dependent (requiring the authors to aggregate across runs as they suggest). I do not see a separate internal inconsistency that would force rejection; the method is clearly presented and the code is released. The conditional verdict is appropriate, with the condition being demonstrated stability across SAE runs.","tokens_in":21510,"tokens_out":6714,"duration_ms":63280,"concrete_test":"Rerun the full BehaviorBox pipeline for the Llama2-13B vs OLMo2-13B comparison at least 10 times, varying only the random seed and data shuffle while keeping all hyperparameters in Table 3 fixed. For each run, record the full set of features surviving the Section 6.1 filters, compute pairwise Jaccard overlap of each feature's top-50 activated words across runs, and note the sign of the median probability difference for the strings tab, quotation mark, https, morrow, period+quote, whitespace, lt, and comma. Then repeat the Section 8 generation experiment separately for each run using that run's eligible string features.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that BehaviorBox's discovered features are stable enough to support generalizable statements about model differences. The full pipeline is run once per model pair, and every reported feature list in Section 7, every dendrogram, and the eight string hypotheses tested in Section 8 are products of that single SAE initialization and data ordering. Appendix A.1 explicitly states that different random seeds and data orderings \"can largely impact the space of features found by SAEs,\" and the authors note variation across runs with only partial overlap. No overlap statistics, seed-sweep results, or consensus-feature analysis are reported. If a different seed yields a different set of salient features, then the qualitative conclusions (e.g., that 13B models excel at long-tail stylized text, or that Llama is better at tabs in dialogue) may be artifacts of one run. The generation experiment does not resolve this, because the tested strings were chosen after inspecting the features of that same single run, and only 8 of many possible string features were tested without correction for selection or multiple comparisons; the \"morrow\" result is also an uncalibrated 20-vs-0 count. Thus the strongest evidence for generalization is not independent of the seed-sensitive feature-selection step. Establishing multi-seed stability is a necessary condition for the central claim to hold as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BehaviorBox, an automated pipeline for discovering fine-grained, human-interpretable features that distinguish the next-token prediction performance of two language models. The method builds a per-word representation by concatenating Longformer contextual embeddings with the probabilities assigned by the two evaluated LMs, trains a sparse autoencoder (BatchTopK SAE) on the resulting 770-dimensional vectors, filters features by coherence and by median probability/log-probability difference, and labels surviving features with an LLM annotator. The authors apply the pipeline to comparisons across model size (7B vs. 13B), post-training (base vs. chat/DPO), and model family (Llama-2 vs. OLMo-2), and report qualitative feature categories such as formatting, dialogue, and vernacular text. They also test eight string-frequency hypotheses in free generation for Llama-13B vs. OLMo-13B, finding six significant results. The paper includes full appendices with hyperparameters, annotation prompts, and all feature tables, and releases code.","tokens_in":21768,"tokens_out":2752,"duration_ms":27314,"significance":"If the central claims hold, BehaviorBox would be a useful complement to corpus-level perplexity and benchmark-based evaluation, offering automated hypothesis generation about where one LM systematically differs from another. The manuscript's strengths include a clearly specified pipeline, a thoughtful treatment of tokenization aggregation, a partially automated labeling procedure with validation, and a free-generation experiment that makes falsifiable predictions. The paper is also transparent about its limitations, particularly in Appendix A.1 regarding SAE sensitivity to random seeds and data ordering. However, the empirical support for the core claim that discovered features are stable and generalizable is currently incomplete, and the selective reporting of features and hypotheses makes several conclusions partly tautological or under-supported. The work is timely and of interest to the evaluation and interpretability communities, but the main claims require additional reproducibility evidence.","major_comments":[{"comment":"Section 6.1 filtering","section":"Section 7 and Appendix A.1"},{"comment":"Section 8","section":"Section 6.1"},{"comment":"Appendix A.1, Table 4","section":"Section 8"},{"comment":"The choice of the probability up-weighting fraction (0.7) is based on a single small-scale sweep over a fraction of the data, and the paper does not assess how sensitive the downstream features and conclusions are to this hyperparameter or to other SAE hyperparameters (dictionary size, k, batch order). Given that Appendix A.1 already notes the variability across runs, a sensitivity analysis that varies the probability weight and reports the overlap or stability of the resulting feature sets would materially strengthen the claim that the method is robust and not tuned to produce particular outcomes.","section":"Appendix A.1, Table 4"}],"minor_comments":[{"comment":"The abstract contains a typo: 'datatset' should be 'dataset'.","section":"Abstract"},{"comment":"The table caption uses 'Olmo-13B' with inconsistent capitalization; elsewhere the paper uses 'OLMo-13B'.","section":"Table 2"},{"comment":"The sentence 'Additionally, while did not have a conclusive result for \"morrow\"' is grammatically incomplete; it should read 'while we did not have a conclusive result'.","section":"Section 8"},{"comment":"Several feature labels contain typos: 'puncutation' (Table 9), 'senteses' (Table 9), and 'preceeding' (Table 10) should be corrected. One label in Table 9 also ends with a stray '<SEP>' token, likely a formatting artifact.","section":"Tables 9 and 10"},{"comment":"The dendrograms are dense and the feature labels are small; a high-resolution version or an interactive supplement would improve readability, especially for the many tab-related labels.","section":"Figures 2-4"},{"comment":"The description of probability up-weighting states that the probability components make up 70% of the total magnitude, but it is not clear whether this rescaling is applied per input vector or globally; clarifying this would aid reproducibility.","section":"Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for the journal's scope and the authors have been transparent about known limitations. The main concern is not the method's basic validity but the mismatch between the strength of the claims (generalizable, automated discovery of behavior differences) and the evidence base (single SAE runs, hand-set thresholds, post-hoc hypothesis selection). If the authors add a seed-stability analysis and temper the claims about generalization, the paper could be acceptable. I would not recommend rejection, as the concerns are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on BehaviorBox. The new thing is real: they take slice finding, which mostly lived in image classification, and adapt it to language modeling by building performance-aware word embeddings (contextual embedding concatenated with both LMs' word probabilities) and then decomposing that space with a sparse autoencoder. The result is an automatic way to generate hypotheses about fine-grained contexts where one LM beats another — e.g., \"conditional 'were' in 'if you were'\" or \"tabs after punctuation in narrative text\" — without needing predefined slices. The paper is clearly written, the code is released, and they apply it across model size, family, and post-training. The generation experiment, where they test eight string-level hypotheses derived from the discovered features, is a good step: six of eight come out significant, which suggests the features are not just artifacts of the analysis corpus.\n\nThe main soft spot is the one the stress-test note flags: all the reported insights come from a single SAE run per model comparison. Appendix A.1 explicitly says different random seeds and data orderings \"can largely impact the space of features found by SAEs,\" and they do not report overlap statistics or a seed sweep. That matters because the qualitative conclusions in Section 7 and the string hypotheses in Section 8 are all products of that one run. Without evidence that the features are stable across seeds, you cannot claim these specific behaviors are robust differences between the models. It's fixable — run the pipeline multiple times, report Jaccard overlaps or consensus features — but until then the main findings are tentative.\n\nTwo smaller issues. First, the filtering step guarantees that kept features have large probability differences (median diff > 0.1 or log-diff > 1), so saying \"these features separate the models\" is partly built into the selection. The coherence and generalization are still empirical, but the framing should be more careful. Second, the generation experiment picks only eight strings, chosen after seeing the same single run, and doesn't correct for multiple comparisons; the \"morrow\" result of 20 vs 0 is uncalibrated. These are minor compared to the seed problem.\n\nWho should read this: anyone doing model comparison or interpretability. It's a useful hypothesis generator, not the final word on performance differences. I'd send this to peer review, but I'd ask for multi-seed stability analysis and a clearer treatment of the selection tautology before publication.","headline":"BehaviorBox is a promising hypothesis-generation tool for LM comparison, but its reported insights currently rest on single-run SAE features that the appendix admits are seed-sensitive.","tokens_in":22293,"tokens_out":2503,"would_cite":true,"duration_ms":22783,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BehaviorBox automatically discovers fine-grained, human-interpretable text features where one language model outperforms another, and shows these features generalize to free generation.","keywords":["language model evaluation","sparse autoencoders","slice finding","performance-aware embeddings","model comparison","perplexity","interpretability","feature discovery"],"falsifier":"Run BehaviorBox on the same model pair and corpus multiple times with different random seeds and data-order shuffles, then compare the resulting feature sets. If the coherent features and their probability-difference signs are not largely reproducible across runs, or if a second run produces contradictory labels such as one model outperforming on tabs in one run and the other in another, the central claim that the discovered features capture stable model differences would be refuted.","tokens_in":21294,"feed_emoji":"🔍","tokens_out":7443,"duration_ms":64718,"temperature":0.7,"pith_summary":"BehaviorBox is an automated pipeline for comparing two language models by locating fine-grained, human-interpretable text features where one model assigns higher probability than the other. The paper argues that corpus-level perplexity hides these differences, and that features such as conditional 'were' after 'if you', tabs in narrative text, or archaic spellings can be discovered bottom-up without pre-specified domains. The method builds a performance-aware embedding for each word by concatenating a contextual embedding with the two models' word probabilities, trains a sparse autoencoder to decompose these representations, and labels the resulting groups with an LLM annotator. Applied to models differing in size, post-training, and family, BehaviorBox surfaces coherent behavioral contrasts, including differences between models with nearly identical perplexity. A generation experiment confirms that six of eight string hypotheses derived from discovered features appear at significantly different rates in free generation, supporting the claim that the features reflect real behavioral differences.","feed_headline":"BehaviorBox auto-discovers where one language model beats another","feed_subtitle":"It finds fine-grained text contexts, from conditional 'were' to tabs, that corpus-level perplexity hides.","key_machinery":"The load-bearing mechanism is the performance-aware contextual embedding: a 770-dimensional vector per word made of a 768-dimensional Longformer word embedding concatenated with the word probabilities assigned by the two compared language models. The probabilities are up-weighted so that their components make up 70% of the input magnitude, ensuring the autoencoder actually uses the performance signal. A sparse autoencoder with a batch top-k operation, where only the top k activations per batch survive, learns a 3000-feature dictionary, and each dictionary element is treated as a candidate behavior feature. The paper selects, for each feature, the words with highest activations, filters features by median probability and log-probability difference thresholds, and uses a strong LLM annotator to label coherent groups, with manual correction by the authors.","core_discovery":"The central discovery is that fine-grained, interpretable differences in language-model behavior can be extracted automatically from a corpus by treating each word as a point in a joint space of context and performance. For every word, BehaviorBox averages the contextual token embeddings from Longformer, multiplies the token probabilities assigned by the two compared models, and concatenates the embedding with the two probabilities into a 770-dimensional vector. A sparse autoencoder with a batch top-k sparsity constraint decomposes these vectors into additive features; the top-activating words for each feature form a slice, and features are kept only if the median probability or log-probability difference between models exceeds fixed thresholds. The result is a list of coherent, labelable slices, such as 'tabs following another tab after sentence-ending punctuation' or 'conditional were in the phrase if you were', that identify where one model is easier or harder to predict than the other. The paper reports that these features generalize beyond the analysis corpus, with six of eight string hypotheses significantly distinguishing free generations of Llama-13B and OLMo-13B.","pith_inferences":["A natural extension is to run multiple sparse-autoencoder initializations and take the union of discovered features, since the paper notes in Appendix A.1 that different random seeds and data orderings can substantially change the feature space; this would test whether the reported contrasts are stable.","The method's word-level unit and probability-based thresholds are not specific to English or to the two model families tested, so it could plausibly be applied to other languages, code-heavy corpora, or multimodal sequence models.","The generation experiment tests only eight hand-selected strings; a stronger test would automatically map discovered features to generation statistics across additional model pairs to see whether the generalization claim holds broadly.","The meta-categories in the qualitative analysis were produced by k-means clustering, LLM labeling, and manual inspection; an automated replication without manual corrections would reveal how much of the reported insight is attributable to the pipeline itself."],"forward_implications":["If BehaviorBox is correct, practitioners can replace hand-specified benchmark slices with automatically discovered behavioral contrasts for any pair of language models.","Features found in corpus evaluation can be turned into testable hypotheses about open generation, as demonstrated by the significant string-frequency differences for six of eight hypotheses.","Perplexity deltas do not predict the number or coherence of salient behavioral differences; near-identical perplexity can coexist with many well-separated features, as in the family comparisons.","Post-trained chat models can outperform their base counterparts on conversational phrases and multiword constructions even while having substantially worse perplexity, so aggregate metrics alone mislead about where each model is stronger."],"supporting_citations":[{"why":"Supplies the Longformer contextual embeddings that form the semantic part of each word's performance-aware representation.","marker":"Beltagy et al., 2020"},{"why":"Defines slice finding, the task of automatically identifying data subgroups where a model underperforms, which BehaviorBox extends to language-model comparison.","marker":"Chung et al., 2018"},{"why":"Domino shows how learned representations plus model predictions can discover systematic errors, the conceptual template for performance-aware embeddings.","marker":"Eyuboglu et al., 2022"},{"why":"Establishes sparse autoencoders as a method for extracting human-interpretable features from language models.","marker":"Cunningham et al., 2023"},{"why":"Provides the batch top-k operation used to enforce sparsity in the autoencoder.","marker":"Bussmann et al., 2024"},{"why":"Provides the Dolma corpus from which the word-level dataset for all comparisons is sampled.","marker":"Soldaini et al., 2024"},{"why":"Provides the nonparametric test used to evaluate whether discovered features produce significantly different string-generation frequencies.","marker":"Mann and Whitney, 1947"},{"why":"Defines the Llama 2 model family whose 7B and 13B base and chat versions are compared.","marker":"Touvron et al., 2023"},{"why":"Defines the OLMo 2 model family whose 7B and 13B base and DPO versions are compared.","marker":"OLMo et al., 2024"},{"why":"Provides the Claude 3.5 Sonnet model used as the automated labeler for the discovered features.","marker":"Anthropic, 2024"}],"fun_headline_variants":["BehaviorBox auto-finds where LMs differ in fine-grained contexts","BehaviorBox digs up hidden text contexts where one LM wins","Automated tool BehaviorBox pinpoints fine-grained LM differences","BehaviorBox reveals fine-grained text features that separate LMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the specific feature sets found by one sparse-autoencoder run represent the true behavioral differences between the two models; the paper itself notes in Appendix A.1 that different random seeds and data orderings can largely change the space of features discovered, so a single run's feature list may be a sample rather than a stable account.","fun_headline_variants_meta":{"raw":{"variants":["BehaviorBox auto-finds where LMs differ in fine-grained contexts","BehaviorBox digs up hidden text contexts where one LM wins","Automated tool BehaviorBox pinpoints fine-grained LM differences","BehaviorBox reveals fine-grained text features that separate LMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000503,"raw_usage":{"total_tokens":2460,"prompt_tokens":948,"completion_tokens":1512,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":1443}},"tokens_in":564,"tokens_out":1512,"duration_ms":11336,"temperature":1.0,"reasoning_tokens":1443,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:28:09.489764+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run BehaviorBox on the same model pair and corpus multiple times with different random seeds and data-order shuffles, then compare the resulting feature sets. If the coherent features and their probability-difference signs are not largely reproducible across runs, or if a second run produces contradictory labels such as one model outperforming on tabs in one run and the other in another, the central claim that the discovered features capture stable model differences would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines slice finding, the task of automatically identifying data subgroups where a model underperforms, which BehaviorBox extends to language-model comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Domino shows how learned representations plus model predictions can discover systematic errors, the conceptual template for performance-aware embeddings."}],"review_version":1}