{"id":"bc7145db-0314-4656-a014-d6703aa5a5d6","arxiv_id":"2506.15617","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper claims that regret in LLMs is encoded by interacting neuron groups detectable in the final hidden layer, using new S-CDI, RDS, and GIC metrics.","lead":"This paper probes how large language models internally encode 'regret' after being shown their own misinformation, proposing new metrics to locate regret layers and neurons in Llama-2 models. It reports an M-shaped decoupling pattern and a compositional regret architecture, but the central metric is internally inconsistent and the probe may only detect the word 'regret'.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Probe labels are confounded with the literal token 'regret' (Appendix D), so the reported 98-99% accuracies and all downstream neuron-group results may only decode word identity, not a regret state.","rationale":"The reader's weakest assumption correctly identifies the probe construction in Appendix D as the load-bearing point. The paper's central claim is that regret is compositionally encoded at the neuron level, evidenced by high probe accuracy, RDS neuron groupings, and GIC intervention results. All three evidences are computed on a probe dataset where positive examples are hidden states at the literal token 'regret' and negative examples are hidden states from a1, which by construction does not express regret and therefore typically does not contain that token. A classifier can reach near-perfect accuracy simply by detecting the token embedding or its local context, which would make the 'regret neurons' and 'compositional architecture' descriptions artifacts of lexical identity rather than of a cognitive state. The paper's own Appendix G.5 defends the anchoring strategy, but its validation arguments (intervention effects, cross-scale consistency, RDS separation) all presuppose that the probe labels correspond to regret states; if the labels are token-identity confounded, those arguments are circular. The proposed concrete test directly removes the confound by using 'regret' tokens in non-regret contexts as negatives. If the probe cannot distinguish regret from non-regret uses of the same token, the central claim fails. The reader's REJECT verdict is therefore well supported, and no adjustment is needed.","tokens_in":21291,"tokens_out":3414,"duration_ms":40129,"concrete_test":"Run a matched-control probe: collect hidden states at the token 'regret' in a2/a3 (positive) and at the token 'regret' in newly generated non-regret sentences using the same surface frames (e.g., 'I regret to inform you...', 'I do not regret...', 'regretfully,...') as negative, matched for position and sentence structure. Train the same 2-layer MLP probe on an equal-size split; if accuracy falls to ~50%, the original 98-99% accuracy was driven by token identity. If it stays high, repeat RDS/GIC on this control; only if intervention effects replicate on non-regret 'regret' tokens is the compositional claim sustained.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the probe dataset. Appendix D states that positive samples are hidden states at the token 'regret' in a2/a3, while negative samples are hidden states at 'equivalent positions in a1 where no regret is expressed.' Initial answers (a1) are misinformation responses that by construction contain no regret expression; hence they generally lack the token 'regret'. The probe can therefore achieve ~98% accuracy by recognizing the lexical item or position 'regret' rather than a regret-specific cognitive state. This confound propagates: RDS (Eq. 7) compares activations at 'regret' vs 'non-regret' tokens, so the 'RegretD' group is identified by whatever features distinguish those tokens; the GIC interventions (Eq. 9, Tables 2-3) then show only that ablating these neurons degrades classification of the token 'regret'. The abstract's claim of 'compositional architecture of regret at the neuron level' is thus unsupported unless the probe separates regret state from token identity. The paper's Appendix G.5 asserts token anchoring is principled and cites validation via interventions, but those interventions inherit the same confound; cross-scale consistency of a confounded signal is not disambiguating. No code or data are released, so the 'equivalent position' matching cannot be checked. This is the load-bearing weak point: if the control is not token-matched, every downstream result (M-shape, neuron categories, compositional GIC) is a statement about the word 'regret'.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-stage prompting pipeline that elicits regret expressions in Llama-2 models after they produce misinformation: fake evidence yields an initial answer, a weak hint yields a second answer, and real evidence yields a third answer with explicit regret. From the collected hidden states, the authors define a Supervised Compression-Decoupling Index (S-CDI) to select the layer where regret is supposedly decoupled, a Regret Dominance Score (RDS) to partition neurons into RegretD, Non-RegretD, and DualD groups, and a Group Impact Coefficient (GIC) to measure the effect of deactivating these groups on probe accuracy. They report an M-shaped S-CDI pattern across layers and a compositional architecture in which combined deactivation of RegretD+Non-RegretD or RegretD+DualD neurons causes large probe-accuracy drops.","tokens_in":21639,"tokens_out":11103,"duration_ms":126559,"significance":"The paper addresses a genuinely interesting question: whether a metacognitive state such as regret has a compositional neural basis in transformer language models. The three-stage elicitation workflow is a reasonable attempt to obtain explicit regret expressions, and the multi-scale comparison across 7B/13B/70B models, the inclusion of random-deactivation baselines, and the reported tau-sensitivity analysis are good methodological practices. If the central claims were supported, the proposed metrics and dataset would be useful contributions to neuron-level interpretability. However, the current evidence does not support those claims: the probe labels are anchored on the surface token 'regret' without controlling for token identity, the S-CDI metric combines lower-is-better and higher-is-better terms in a way that makes its minimum ill-defined, and the GIC 'interventions' modify probe inputs rather than the model's generation behavior. The required controls and behavioral experiments constitute a substantial new study rather than local revisions.","major_comments":[{"comment":"The probe dataset is confounded with the literal token 'regret'. Positive samples are hidden states at the token 'regret' in a2/a3, while negative samples are hidden states at 'equivalent positions' in a1, which by construction express no regret and therefore generally do not contain the token 'regret'. A classifier can reach the reported 98-99% accuracy by recognizing the surface word or its positional pattern rather than a regret state. This confound propagates to RDS (Eq. 7), which compares activations at 'regret' versus non-'regret' positions, and to the GIC results (Tables 2-3), which evaluate probe accuracy on those same token-labeled states. The validation arguments in Appendix G.5 inherit the same confound, and the 'equivalent position' matching cannot be checked because no code or data are released. The authors should add token-matched negative controls (for example, non-regret uses of the word 'regret'), test generalization to regret paraphrases that do not contain the literal token, and report generation-level outcomes; without these, the central claim of a compositional regret architecture is unsupported.","section":"Appendix D; Sections 3.1 and 4.2"},{"comment":"The S-CDI metric is directionally inconsistent. The paper states that lower CDI means better compression and that lower Ie means better inter-class separation, yet the class-separability factor in Eq. (1) is Ic/(1-Ie), which increases as Ie increases and also increases as Ic increases. Multiplying a lower-is-better compression term by a higher-is-better separability term means that the minimum of S-CDI is not a well-defined optimum. The claim in Section 4.2 that the last layer has the 'lowest S-CDI values, where regret signals are optimally decoupled' is therefore not justified. The authors need to redefine S-CDI so that both components point in the same direction, or validate the resulting layer ranking against an external criterion independent of the probe.","section":"Section 3.2, Eq. (1)-(6)"},{"comment":"The GIC experiments are not interventions on the model. Deactivating a neuron group means setting those activations to -1 in the hidden states that are then passed to the probe; the model's generation is never modified. The observed probe-accuracy drops therefore show only that the probe relies on those neurons, and the same hidden states are used both to define the neuron groups and to evaluate them, creating a circular evaluation. The Conclusion's statement that targeted neuron interventions 'confirm their causal role' is not supported. The authors should perform decoding-time ablation or activation patching and measure changes in the generated text, with random ablations of matched size as controls.","section":"Section 3.2, Eq. (9); Tables 2-3; Conclusion"},{"comment":"The M-shaped decoupling pattern is not established by the reported data. The 7B model is evaluated at only five layers (1, 8, 16, 24, 32), the 13B model at eight layers, and the 70B model at fourteen layers. Connecting these sparse points in Fig. 3 cannot resolve a four-phase pattern of coupling, decoupling, coupling, and decoupling. The authors should report all layers, or at least a dense uniform sample with confidence intervals, before making this claim.","section":"Section 4.1; Table 1; Fig. 3"}],"minor_comments":[{"comment":"The sentence 'But there has been no quantitative analysis on which layers are the most important' is ungrammatical and should be revised.","section":"Section 2"},{"comment":"RDS assumes paired regret and non-regret instances, but the pairing and alignment of positions across a1, a2, and a3 is not defined; please clarify how (Zr)_ik and (Zn)_ik are matched and how negative activation values are handled.","section":"Section 3.2, Eq. (7)"},{"comment":"The random perturbation protocol is not described; specify how many neurons are removed, whether removals are performed per layer, and how many random seeds are used.","section":"Table 1"},{"comment":"The manuscript does not include a code or data availability statement; given that the 'equivalent position' matching and dataset construction are load-bearing, this must be added.","section":"Overall"},{"comment":"The claim of 'up to 50.7% degradation' is not consistent with Table 3, where the 70B accuracy drops from 99.7% to 49.3%, a decrease of 50.4 percentage points; please state whether the F1-score is intended.","section":"Conclusion"},{"comment":"The radar chart caption refers to 'a critical parameter tau' where model scale appears to be meant; the wording is confusing and should be corrected.","section":"Appendix F, Fig. 7"}],"recommendation":"reject","confidential_remarks":"The token-anchoring confound is visible from the manuscript text alone, and the absence of code or data makes the 'equivalent position' matching impossible to audit. The S-CDI direction issue and the non-causal nature of the GIC 'interventions' are further load-bearing problems. I would not oppose a fresh submission if the authors add token-matched controls, generation-level behavioral interventions, and a corrected metric, but the current manuscript does not support its central claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the quick read on 2506.15617. The paper asks a genuinely new question—where and how regret is encoded at the neuron level in LLMs—and builds a three-stage dataset that elicits regret through fake evidence, hints, and real evidence. That workflow is the strongest part, and the cross-scale analysis on Llama-2 7B/13B/70B is a reasonable effort. If the method were sound, the compositional story would be interesting.\n\nThe method is not sound. The central probe is confounded by token identity. Appendix D defines positive samples as hidden states at the token 'regret' in answers 2 and 3, and negative samples as 'equivalent positions' in answer 1, which by construction contains no regret and, in practice, no 'regret' token. A probe can hit 98–99% simply by recognizing the word 'regret'. That confound propagates into RDS (activations at 'regret' vs non-regret tokens), into the neuron groupings, and into the GIC interventions, which only show that ablating certain neurons hurts classification of the token. The Appendix G.5 defense—that the hidden state at a token captures context—does not solve the missing token-matched control. The paper releases no code or data, so we can't verify the 'equivalent positions' matching.\n\nThere's also an internal metric issue. S-CDI is defined as CDI times Ic/(1-Ie). CDI is lower-is-better; the class-separability factor is higher-is-better (higher Ic, lower Ie). The paper then picks the layer with the lowest S-CDI and calls it 'optimally decoupled.' That's inconsistent—minimizing the product doesn't minimize a decoupling criterion unless both terms point the same direction.\n\nOn the plus side, the dataset construction and the three-way neuron categorization are a plausible starting point for studying meta-cognitive states, and the heuristics in the appendices are honestly labeled as speculative. The writing is clear. But the load-bearing claim about a compositional architecture is unsupported as presented. This needs major revision: token-matched controls, a corrected index, and interventions measured on model outputs rather than probe accuracy.\n\nVerdict: reject the central claims as is, but send to peer review—the question is worth referee time and the flaws are fixable in principle.","headline":"The paper asks a fresh question about where regret lives in LLM neurons, but the probe is confounded by the literal token 'regret' and the central metric is internally inconsistent, so the compositional architecture claim does not hold as presented.","tokens_in":22165,"tokens_out":3900,"would_cite":false,"duration_ms":40551,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Regret in Llama-2 models is a compositional, group-level neural code, not a single-neuron feature.","keywords":["regret","large language models","neuron interpretability","probing","compositional encoding","misinformation correction","supervised compression-decoupling index","neuron intervention"],"falsifier":"Run the identical probe and intervention pipeline on hidden states from the token \"regret\" in matched non-regret contexts (e.g., \"I regret to inform you...\" or \"she regrets...\") at the same layers; if classification stays near 98–99 percent, the signal is token-driven rather than regret-driven, and if deactivating the same neuron groups causes the same collapse, the compositional claim is tied to the word, not the state.","tokens_in":21117,"feed_emoji":"🧠","tokens_out":8636,"duration_ms":85723,"temperature":0.7,"pith_summary":"This paper argues that regret, as expressed by Llama-2 chat models after they are confronted with evidence correcting their own misinformation, is not stored in any single neuron but in the coordinated activity of neuron groups. To show this, the authors construct a three-stage dataset in which models first answer from fake evidence, then receive a weak hint, then receive real evidence and say \"regret\"; train probes on hidden states at that token; and use three new metrics to locate the layer where regret signals separate most cleanly and to group neurons by their regret dominance. The central finding is a compositional architecture: deactivating only regret neurons, only non-regret neurons, or only dual neurons leaves probe accuracy high, while deactivating regret neurons together with dual neurons (or with non-regret neurons) drops accuracy from near 98 percent to as low as 49 percent. If correct, the paper establishes that a meta-cognitive state such as regret can be located and manipulated at the neuron-group level in large language models, with implications for reliability and for interpreting how cognitive states are distributed across transformer layers.","feed_headline":"Regret in LLMs is coded by cooperating neuron groups, not one neuron","feed_subtitle":"Removing regret-plus-dual neurons together cuts detection accuracy to about half; single-group removals barely matter.","key_machinery":"The machinery is a pipeline running on hidden states from Llama-2 transformer layers. The dataset is generated by a three-stage prompting protocol (fake evidence, weak hint, real evidence) that produces paired regret and non-regret answers. S-CDI (Supervised Compression-Decoupling Index) multiplies a compression-efficiency term by a class-separability ratio to score each layer, selecting the final layer as the optimal decoupling point. RDS (Regret Dominance Score) computes, per neuron, the ratio of its activation on regret instances to its activation on regret-plus-non-regret instances, then uses mean and standard deviation thresholds to split neurons into RegretD, Non-RegretD, and DualD groups. GIC (Group Impact Coefficient) deactivates one or more groups by setting their activations to −1 and compares probe accuracy after deactivation to baseline and to the average of single-group effects. The measurement instrument throughout is a two-layer MLP probe trained on hidden states at the token \"regret\" in second and third answers (label 1) versus equivalent positions in first answers (label 0).","core_discovery":"On the paper's own terms, the discovery is that regret in Llama-2 models has a compositional, group-level neural code. Using the Supervised Compression-Decoupling Index (S-CDI), the authors find that the lowest S-CDI value occurs at the final transformer layer, where regret representations are optimally decoupled from context; across layers S-CDI traces an M-shaped pattern of coupling and decoupling that they attribute to attention's context integration near the top of the network. Within that optimal layer, the Regret Dominance Score (RDS) divides neurons into regret-dominant (RegretD), non-regret-dominant (Non-RegretD), and dual (DualD) groups. The Group Impact Coefficient (GIC) then shows the load-bearing result: removing either RegretD or DualD neurons alone barely changes a trained probe's 98–99 percent accuracy in detecting regret hidden states, but removing them together collapses accuracy to 49.3–63.2 percent, with the strongest collapse in the 70B model. The paper interprets this as regret emerging from the cooperation of functionally distinct neuron groups rather than from any individual group.","pith_inferences":["The method's token anchor means the paper establishes overt regret only; a natural extension is to test whether implicit regret (reversing an answer without saying \"regret\") shares the same neuron groups or recruits a different circuit.","The scaling result suggests a testable prediction: fine-tuning or RLHF on correction-heavy data should strengthen RegretD-DualD mutual information rather than simply increasing single-neuron selectivity.","If the compositional architecture is general, interventions to suppress or induce apology-like behavior in deployed models would need to target the RegretD+DualD intersection, an implication the paper does not pursue for safety applications.","Because S-CDI chooses a layer by minimizing a compression/separability tradeoff, the metric could be lifted to other meta-cognitive states (doubt, uncertainty, avoidance) to test whether the M-shaped decoupling pattern is a general transformer feature."],"forward_implications":["Neuron editing for regret must be group-level: single-group deactivation leaves probe accuracy essentially unchanged, while RegretD paired with DualD or Non-RegretD cuts it to 49–63 percent.","The final transformer layer is the most promising place to read or modify regret states, because S-CDI identifies it as the layer where regret is cleanest from context.","The M-shaped S-CDI pattern predicts that the upper, pre-final layers are a context-integration phase; interventions that disrupt attention in those layers should weaken regret decoding even if the final-layer groups are left intact.","The compositional effect grows with scale: the 70B model shows the strongest RegretD+DualD coupling (mutual information 0.071) and the largest intervention collapse, so larger models are not simply bigger versions of smaller ones for regret."],"supporting_citations":[{"why":"Supplies the conflictQA-popQA-gpt4 seed data and the knowledge-conflict setup from which the regret dataset is built.","marker":"[59]"},{"why":"Provides the layer-wise probing paradigm and the evidence that attention states are chaotic near higher layers, which motivates S-CDI layer selection and the M-shape interpretation.","marker":"[16]"},{"why":"Contributes the Modality Dominance Score that RDS adapts into a regret dominance grouping.","marker":"[33]"},{"why":"Gives the information-bottleneck rationale for S-CDI's compression-versus-task-relevant-information balance.","marker":"[65]"},{"why":"Supplies the variational information bottleneck formulation that S-CDI extends with a supervised class-separability term.","marker":"[64]"},{"why":"Establishes sparse feature circuit discovery and intervention methods that GIC extends from individual circuits to group-level impact.","marker":"[56]"},{"why":"Defines the Llama-2 architecture and model family whose hidden states are probed and intervened on.","marker":"[67]"}],"fun_headline_variants":["Regret in LLMs is group-coded, not single-neuron","Removing regret plus dual neurons halves detection accuracy","LLM regret: dual and regret neurons must cooperate","Neuron groups, not singles, encode LLM regret","Compositional code: regret emerges from neuron group interplay"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the hidden state at the literal token \"regret\" in second and third answers faithfully anchors the regret mechanism, with the matching position in the first answer serving as a clean non-regret control; if the probe is instead detecting the word or its position, the layer selection, neuron grouping, and intervention results are not about regret as a cognitive state.","fun_headline_variants_meta":{"raw":{"variants":["Regret in LLMs is group-coded, not single-neuron","Removing regret plus dual neurons halves detection accuracy","LLM regret: dual and regret neurons must cooperate","Neuron groups, not singles, encode LLM regret","Compositional code: regret emerges from neuron group interplay"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000684,"raw_usage":{"total_tokens":3152,"prompt_tokens":1044,"completion_tokens":2108,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":2027}},"tokens_in":660,"tokens_out":2108,"duration_ms":17815,"temperature":1.0,"reasoning_tokens":2027,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:52:47.615987+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical probe and intervention pipeline on hidden states from the token \"regret\" in matched non-regret contexts (e.g., \"I regret to inform you...\" or \"she regrets...\") at the same layers; if classification stays near 98–99 percent, the signal is token-driven rather than regret-driven, and if deactivating the same neuron groups causes the same collapse, the compositional claim is tied to the word, not the state.","supporting_citations":[{"cited_title":"Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts","cited_arxiv_id":null,"evidence_quote":"Supplies the conflictQA-popQA-gpt4 seed data and the knowledge-conflict setup from which the regret dataset is built."},{"cited_title":"Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models","cited_arxiv_id":"2502.14888","evidence_quote":"Contributes the Modality Dominance Score that RDS adapts into a regret dominance grouping."},{"cited_title":"Compressing neural networks using the variational information bottleneck","cited_arxiv_id":null,"evidence_quote":"Supplies the variational information bottleneck formulation that S-CDI extends with a supervised class-separability term."}],"review_version":1}