{"id":"f30830d3-415f-4691-b9a8-800b4c3ea99e","arxiv_id":"2505.22630","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLMs systematically combine abstract category cues from a query with features from irrelevant context, causing structured answer flips, a behavior the authors call class-based (mis)generalization.","lead":"Adding an irrelevant sentence before a factual question makes large language models change their answer in a predictable way, by blending the question's category (like language) with a feature from the extra sentence (like Japan). The authors call this class-based (mis)generalization and show through neural probes that two competing internal circuits decide which answer wins.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The behavioral evidence for class-based generalization is undermined by selection: candidates are defined by Eq. 3 (answers changed by context) with no null baseline, so the 71% annotation and PMI ≈ 4 may only reflect context-sensitivity, not a class+feature mechanism.","rationale":"The most load-bearing step is the inference from observational behavioral statistics to a structured class+feature mechanism. The paper's strongest quantitative evidence—71% annotation and mean PMI ≈ 4—is produced after conditioning on Eq. 3, which selects candidates by the very property the paper aims to explain (context-induced change). This makes the statistics compatible with a much weaker claim: that LLM predictions are context-sensitive. A permutation or control-condition baseline is necessary to show the specific structure. I do not find the mechanistic results sufficient to rescue the claim on their own: logit attribution and activation patching are consistent with the hypothesis but also with simpler distributed priming, and the knockout flips only a minority of cases. This is not an accusation of circular reasoning in a formal sense—the paper does not claim the PMI is a mechanistic test—but the omission of appropriate baselines means the central claim is currently underdetermined. The paper does have genuine strengths: a large controlled dataset, three model families, 39 relation types, and an explicit mechanistic investigation with causal interventions. These are real evidence that irrelevant context can shift predictions and that models track some class-like information, but they do not pin down the proposed class+feature combination mechanism without the missing controls. The proposed tests (permutation PMI and annotated controls with agreement) would, if they survive, materially strengthen the paper; if they do not, the behavioral evidence would be non-diagnostic. I therefore keep the reader's CONDITIONAL verdict, since the concern is addressable in revision.","tokens_in":26517,"tokens_out":12763,"duration_ms":156067,"concrete_test":"Re-run the §5.3 PMI analysis with a permutation null: for each expected class, randomly shuffle the pairing between the 100 contexts and their observed context-based candidates, recompute the mean PMI, and repeat 10,000 times. If the observed mean PMI (~4) does not lie in the upper tail of the permuted distribution, the reported 'strong association' is a selection artifact of Eq. 3 rather than evidence for class-based generalization. As a complementary check on the §5.2 headline, have a second annotator blind-label the same 500 items plus 500 controls sampled from the Q-only top-3, and report Cohen's kappa and the control rate; the 71% figure needs to exceed the control rate substantially to be diagnostic.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"To establish class-based (mis)generalization, the paper must show that context-changed answers come from combining the query's abstract class with a context feature, rather than generic contextual priming. The behavioral evidence in Sections 5.2-5.3 is built on the context-based candidate set defined in Eq. 3: Ccand = top-3(C+Q) \\ top-10(Q). This set contains exactly the answers whose appearance depends on the added context. The 71% annotation then labels these candidates as context-influenced and class-matching; the PMI test in Eqs. 5-6 measures association between context C and its paired Ccand. Both procedures re-detect the selection criterion: any context-sensitive mechanism, including simple lexical association (e.g., 'Honda' → 'Japan'/'Japanese'), will produce high PMI and frequent 'context-influenced, correct-class' labels inside this set. No null model is provided (e.g., candidates from Q-only, shuffled context, or permuted context-candidate pairs), and no inter-annotator agreement is reported for the 71%. Consequently, the headline behavioral statistic does not discriminate the proposed mechanism from weaker priming. The mechanistic section offers some independent evidence, but the attention knockout flips only 225-470 of 1000 datapoints, so it alone cannot carry the claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes that LLM errors induced by irrelevant context follow a structured mechanism called class-based (mis)generalization: the model first derives an abstract class from the query (e.g., 'language'), then selects a feature from the context or query (e.g., 'Japan'), and combines them to produce an answer (e.g., 'Japanese'). The authors define context-based and query-based candidates in Eqs. (3)–(4), report behavioral results on 106M controlled context+query pairs, annotate 500 context-based candidates (claiming 71% combine context features with the correct class), and provide a PMI-based statistical test. They then present mechanistic evidence from logit attribution, activation patching, and attention knockout on Llama-3, Mistral, and Pythia, claiming hierarchical class-to-instance construction and competing context/query circuits. They also test larger models and prompt-template variations, finding the phenomenon persists.","tokens_in":26805,"tokens_out":3800,"duration_ms":46349,"significance":"If the claims hold, the paper would provide a nuanced counterpoint to the 'stochastic parrot' view: LLM errors are not unstructured surface stitching but show systematic class-level generalization, albeit unreliable. The paper's strengths include a large controlled dataset, multiple model families, a public code release, and a triangulation of behavioral and mechanistic methods. However, the central behavioral evidence currently rests on a candidate-selection procedure that may make the headline statistics partly self-fulfilling, and the mechanistic analysis requires stronger validation against null interventions. The idea is timely and the framework is potentially useful, but the evidence as presented is not yet sufficient to establish the mechanism.","major_comments":[{"comment":"The PMI test is partially circular. Ccand is defined in Eq. (3) as a candidate that appears in top-3 under C+Q but not in top-10 under Q, so C and Ccand are selected specifically for their dependence. Any context-sensitive mechanism, including simple lexical priming, would yield positive PMI. The paper needs a null model—for example, permuted context–candidate pairs, candidates sampled from Q-only predictions, or a comparison with query-based candidates—to show that the observed PMI exceeds what the selection rule alone produces.","section":"§5.3, Eqs. (3)–(6)"},{"comment":"The 71% annotation figure lacks inter-annotator agreement and a baseline. The candidates are preselected to be context-dependent, so labeling them 'context-influenced' may simply restate the selection criterion. The authors should report IAA and annotate a control set, such as query-based candidates or random predictions from Q-only, to show that the class+feature combination rate is specific to context-changed answers rather than a general property of top predictions.","section":"§5.2, Appendix D"},{"comment":"The PMI estimation is underspecified. The text says each context Ci is paired with 'its corresponding candidate Ccand,i regardless of the 100 queries,' but Eq. (3) defines Ccand as a function of both C and Q. It is unclear how one candidate per context is obtained when the same context is paired with many different queries. Additionally, the one-sample t-test against E[PMI]=0 is not a natural null for dependence; a permutation or bootstrap test over context–candidate pairings would be more appropriate.","section":"§5.3, Eqs. (5)–(6)"},{"comment":"The critical layers (e.g., 17 and 24 for Llama) are identified from activation-patching maps and then used for attention knockout on the same 1000 datapoints, without a held-out split. The random-layer controls in Appendix H also show nontrivial probability changes in some conditions (e.g., '2 Low' reduces query probability by 7.5 in Llama query-dominant cases, close to the 8.4 reduction for layers 17+24). The mechanistic claim would be stronger with a clearly held-out validation, a statistical comparison of flip rates against random-layer and random-head baselines, and reporting of variance across runs.","section":"§6.2–6.3, Tables 5, 11–13"}],"minor_comments":[{"comment":"The text states 'A mean PMI of around 70' but Table 10 reports mean PMI values of 3.9, 3.7, and 3.8; this should be corrected to 'around 4' or similar.","section":"Appendix E, Table 10"},{"comment":"The caption says 'The first and second row correspond to R1T,l, and the second row is R2T,l' which is confusingly phrased and appears to contain a typo; the intended distinction between the two residual-stream rows should be stated clearly.","section":"Table 4 caption"},{"comment":"The notation h0_{i*,l*} ∪ h1_{-i*,-l*} is not formally defined; the union of a restored activation with all corrupted activations should be described with explicit indexing.","section":"Eq. (11)"},{"comment":"The abstract says '70%' while the body reports 71.0%; these should be reconciled for consistency.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's main risk is methodological rather than conceptual: the behavioral statistics are constructed so that the selection rule for context-based candidates guarantees dependence. If the authors add a proper null model, annotation baselines, and IAA, the central claim could become publishable. The mechanistic section is interesting but would also benefit from held-out validation of the chosen layers. No concerns about novelty or scope; the paper fits the journal's interests."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is worth a serious look: it proposes a concrete mechanism for a known failure mode and backs it with three mechanistic methods on three model families. The central claim—that irrelevant context shifts predictions via an abstract class plus a selected feature—is clearly framed and testable. The logit-lens finding that class tokens rise before instance tokens, and the activation-patching maps showing distinct context and query circuits, are genuinely new evidence. The attention-knockout flips are a real attempt at causal validation, though they only flip 225–470 of 1000 cases. The code is available, which is a plus.\n\nThe soft spots are in the behavioral validation, and they are not minor. The context-based candidate definition (Eq. 3) selects exactly the answers that appear under C+Q but not under Q, so the PMI test re-detects the selection rule. A lexical association account (e.g., 'Honda' → 'Japan') would also produce high PMI. No null model or permutation baseline is given. The 500-annotation 71% figure lacks inter-annotator agreement and a comparison to query-based or random candidates, so it cannot discriminate class+feature from generic contextual priming. The paper acknowledges some limitations, but not this circularity.\n\nThe mechanistic section partly rescues the paper. The two circuits exist in both dominant cases, and the knockout effect is concentrated in the identified layers, which is not explained by a purely surface-level story. But the evidence is suggestive, not conclusive; the knockout flips a minority of cases, and the 'competition' is inferred from average restoration scores.\n\nThis is a solid conditional-accept paper if the authors add proper baselines: annotate query-based candidates as a control, report agreement, replace the PMI t-test with a permutation test against randomly paired contexts and candidates. The central hypothesis may well be right, but the current behavioral numbers don't establish it.\n\nI would send this to a serious referee; it has enough new mechanism and enough honest reporting to deserve full review. I'd bring it to reading group as a case study in how selection effects can creep into behavioral claims. My own verdict is conditional: the machinery is interesting, the proof is incomplete.","headline":"A genuinely new mechanistic story about how irrelevant context shifts LLM predictions, but the behavioral evidence is partly circular and needs null baselines before the headline claim is established.","tokens_in":27352,"tokens_out":2245,"would_cite":true,"duration_ms":26738,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLM errors under irrelevant context are structured: the model derives an abstract class from the query, selects a context or query feature, and combines them to produce an answer.","keywords":["class-based generalization","irrelevant context hallucination","mechanistic interpretability","logit attribution","activation patching","attention knockout","stochastic parrots","factual recall"],"falsifier":"Construct a null distribution for the PMI between contexts and their context-based candidates by randomly permuting the context–candidate pairs within each query class; if the observed mean PMI is not clearly above that null, the claimed context–candidate association is an artifact of selection. Alternatively, hold the context fixed and vary only the abstract class demanded by the query; if the model keeps producing context-derived answers even when the class and the context features are mismatched, the class-selection step is not doing the causal work.","tokens_in":26300,"feed_emoji":"🦎","tokens_out":8182,"duration_ms":81777,"temperature":0.7,"pith_summary":"This paper asks whether the errors large language models make are noise or structure. In a controlled setting, the authors give a model a query together with an irrelevant context—a sentence about Honda, say, before a question about the original language of a film—and show that predictions shift roughly 38–57% of the time across the models tested. They propose that these shifts are the visible trace of class-based (mis)generalization: the model first derives an abstract class from the query (e.g., 'languages'), then selects a feature from the context or the query (e.g., Japan), and combines the class with the feature to produce an answer (e.g., 'Japanese'). Human annotation of 500 shifted answers finds 71% combine an identifiable context feature with the correct abstract class. If the claim holds, it challenges the simple 'stochastic parrot' picture: LLMs are not just stitching together memorized surface forms; they generalize over abstract categories in a systematic but unreliable way, a behavior the authors call stochastic chameleons.","feed_headline":"LLM hallucinations follow a class-based recipe","feed_subtitle":"Irrelevant context makes LLMs pair an abstract category with a context feature, so their wrong answers follow a pattern.","key_machinery":"The central object is the class-based (mis)generalization hypothesis, made testable through the definition of query-based vs. context-based candidate sets: an answer is context-based if it enters the top three under C+Q without being in the top ten under Q alone. The hypothesis says the model first derives an abstract class from the query (e.g., 'languages'), then selects a feature from the context or query (e.g., Japan), and combines class with feature to produce the answer (e.g., 'Japanese'). The load-bearing evidence is the two-circuit competition revealed by activation patching: a query circuit transferring query-subject information from around layer 8 and a context circuit transferring context-subject and context-object information from around layer 17, with the final prediction decided by their relative strength in layers 17–24. Logit attribution supplies the class-to-instance trace: abstract class tokens are promoted in lower layers and concrete candidate tokens in higher layers, so the mechanism is visible in the residual stream.","core_discovery":"The central claim is that irrelevant context hallucinations are not arbitrary errors but the output of a structured two-step mechanism. Given a context–query pair, the model constructs an abstract class representation appropriate to the query ('language', 'place', 'company') in the lower transformer layers, then selects a feature—either from the query (France) or from the irrelevant context (Japan)—and combines class and feature into a candidate answer (French or Japanese). The paper defines query-based candidates as top-three answers under context+query that also appear in the top ten under the query alone, and context-based candidates as top-three answers that do not; predictions are query-dominant or context-dominant depending on which candidate wins. Logit attribution shows the abstract class tokens rising first and concrete candidate tokens rising later, with context- and query-candidate logits forming two competing groups that resolve around layers 17–24. Activation patching locates separate context and query circuits whose relative strength tracks which candidate wins, and attention knockout at the two critical layers flips a large share of predictions in both directions. The authors conclude that the machinery behind these errors is a genuine, if unreliable, form of generalization over abstract classes.","pith_inferences":["If the two-circuit account is correct, targeted interventions at layers 17–24 should reduce irrelevant-context hallucinations, which the paper leaves as future work; one could test whether such interventions generalize beyond the ParaRel relations.","The class-plus-feature mechanism suggests a measurable prediction for prompt design: queries with an ambiguous required class should show larger answer-change rates under irrelevant context than queries with a clear class, a test the paper does not report.","The same class-plus-feature combination may underlie some few-shot demonstration effects: demonstrations provide context features that the class-selection step then incorporates, extending the mechanism beyond hallucination to in-context learning.","Automating the annotation of class and feature combination would let the 71% estimate be checked at scale and across relation types outside the 39 ParaRel relations used here."],"forward_implications":["A large share of wrong answers under irrelevant context is predictable: given the query's abstract class and the features available in the context, one can anticipate what the model will say.","Scaling up does not fix the problem: Pythia 12B and Llama-3 70B show class-based shifts at similar rates to the 7–8B models.","Rewording the query with alternative ParaRel templates leaves the phenomenon essentially unchanged.","The competition between query and context circuits is causally active: knocking out attention in layers 17–24 flips hundreds of predictions per thousand in both dominant directions.","Abstract class construction happens even without added context (Q-only), so class-to-instance refinement is a general feature of factual recall, not an artifact of the irrelevant context."],"supporting_citations":[{"why":"Supplies the ParaRel dataset of 39 factual recall relation types and the prompt templates used for both queries and contexts.","marker":"(Elazar et al., 2021)"},{"why":"States the stochastic parrot critique that the paper's class-based generalization claim refines in the discussion.","marker":"(Bender et al., 2021)"},{"why":"Provides the logit lens method used to trace abstract class logits and candidate logits across layers.","marker":"(nostalgebraist, 2020)"},{"why":"Supplies the activation-patching setup and noise level used to locate the context and query circuits.","marker":"(Meng et al., 2022)"},{"why":"Provides the activation-patching framework the restoration runs are based on.","marker":"(Ghandeharioun et al., 2024)"},{"why":"Supports reading predictions by projecting residual states onto the vocabulary space.","marker":"(Geva et al., 2022)"},{"why":"Baseline showing LLMs are easily distracted by irrelevant context, which this paper mechanistically explains.","marker":"(Shi et al., 2023)"}],"fun_headline_variants":["LLM hallucinations follow a class-based recipe","Structured mistakes: LLMs misgeneralize by class","Stochastic chameleons: LLM errors are class-based","LLM wrong answers reveal class-based reasoning","Context cues trigger class-based LLM errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The statistical validation assumes that an answer appearing only after context is added is the product of a class-based feature-selection mechanism, but such answers are selected precisely because they are context-sensitive, so the PMI test partly re-detects the criterion used to define the candidate set.","fun_headline_variants_meta":{"raw":{"variants":["LLM hallucinations follow a class-based recipe","Structured mistakes: LLMs misgeneralize by class","Stochastic chameleons: LLM errors are class-based","LLM wrong answers reveal class-based reasoning","Context cues trigger class-based LLM errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000253,"raw_usage":{"total_tokens":1607,"prompt_tokens":1028,"completion_tokens":579,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":505}},"tokens_in":644,"tokens_out":579,"duration_ms":5965,"temperature":1.0,"reasoning_tokens":505,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:02:25.969541+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a null distribution for the PMI between contexts and their context-based candidates by randomly permuting the context–candidate pairs within each query class; if the observed mean PMI is not clearly above that null, the claimed context–candidate association is an artifact of selection. Alternatively, hold the context fixed and vary only the abstract class demanded by the query; if the model keeps producing context-derived answers even when the class and the context features are mismatched, the class-selection step is not doing the causal work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the activation-patching framework the restoration runs are based on."}],"review_version":1}