{"id":"e256afa7-70c3-4e80-ac6d-12772a8df6d7","arxiv_id":"2606.20814","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Empirical study finds that pre-fine-tuning model activations predict post-fine-tuning alignment scores and that activation deltas show moderate-to-high subspace overlap between training and evaluation data.","lead":"The paper empirically studies emergent misalignment in language models by examining training loss, pre-fine-tuning activations, and activation shifts between train and evaluation prompts. These correlations could help predict misalignment risks from narrow fine-tuning without exhaustive testing.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"The reader's identification of generalization beyond tested models/datasets/prompts as the weakest assumption matches the only visible uncertainty. No stronger, more technical load-bearing flaw (e.g., in training dynamics, subspace overlap computation, or loss-alignment linkage) is detectable without additional detail, so the UNVERDICTED verdict stands.","tokens_in":1743,"tokens_out":262,"duration_ms":13117,"concrete_test":"Recompute the reported prediction correlations (e.g., via the same regression or similarity metric) after adding a new model family and a disjoint set of 20+ evaluation prompts; if R² or Spearman ρ falls below 0.3 on the new data while remaining high on the original set, the predictive claim is condition-specific.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that pre-fine-tuning prompt activations predict post-fine-tuning alignment scores—rests on observed correlations whose robustness cannot be challenged from the provided abstract alone. No internal inconsistency, missing control, or unjustified assumption is visible in the stated results. The reader's weakest assumption correctly flags generalization risk, but that is an external-validity issue rather than a flaw in the argument as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript investigates emergent misalignment (EM) arising from narrow fine-tuning of language models. It examines three components: (1) relationships between in-domain training loss and out-of-domain alignment scores across datasets and model families, including attempts to find alternative local minima via learning schedules; (2) statistical differences between pre-trained and fine-tuned alignment score distributions, plus the predictive utility of prompt-only activations from pre-trained and instruct models for post-fine-tuning alignment scores; (3) activation deltas showing moderate-to-high subspace overlap and similarity between training and evaluation prompts, with overlaps correlating to shift similarities (controlled against random vectors).","tokens_in":1821,"tokens_out":650,"duration_ms":20708,"significance":"If the reported correlations and predictive relations hold under scrutiny, the work could offer practical signals for anticipating misalignment from pre-fine-tuning activations and data properties, which would be of interest to alignment research. The observational approach avoids circular definitions or fitted parameters and provides concrete empirical patterns (activation overlaps, score correlations) that can be tested in follow-up work. Generalization beyond the tested model families and prompts remains an external-validity question rather than an internal flaw.","major_comments":[{"comment":"Abstract and results on statistical comparisons: the claims that means and standard deviations 'are usually statistically different' and that there are 'some potential signals on overall positive correlation' lack any description of the tests performed, sample sizes per comparison, p-value thresholds, or corrections for multiple testing. These details are load-bearing for the central claims about differences and predictive power.","section":"Abstract / statistical results"},{"comment":"The strongest claim—that evaluation prompt-only activations from pre-trained and original instruct models predict fine-grained alignment scores after narrow fine-tuning—requires explicit reporting of the prediction procedure (feature extraction, regression or classifier used, cross-validation, effect sizes such as R² or AUC). Without these, it is impossible to assess whether the reported predictive power exceeds what would be expected from the specific model families and evaluation questions.","section":"Results on model priors / activation prediction"},{"comment":"Subspace overlap analysis: while a control against random vectors is mentioned, the manuscript should specify the exact metric for overlap (e.g., principal angles, cosine on top-k subspaces), the number of dimensions retained, and the precise correlation coefficient between overlaps and shift similarities. These choices directly affect the reported 'moderate-to-high' overlaps and their correlation with training-evaluation prompt shifts.","section":"Activation delta / subspace overlap results"}],"minor_comments":[{"comment":"The abstract contains informal phrasing ('we tried to induce', 'we first explored') that should be revised to declarative scientific language for consistency with journal style.","section":"Abstract"},{"comment":"Notation for activation deltas and subspace quantities should be defined explicitly (e.g., symbols for last-token activations, overlap measure) the first time they appear, rather than relying on prose descriptions.","section":"Methods / results on activations"}],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments, which help clarify the presentation of our empirical findings on emergent misalignment. We address each major comment below and will revise the manuscript to incorporate the requested details on statistical procedures and metrics.","responses":[{"response":"We agree that the statistical details were insufficiently specified. In the revised manuscript we will report: (i) the exact tests used (two-sided t-tests for means and F-tests for variances, with Wilcoxon rank-sum as robustness check), (ii) sample sizes (N=12 model variants per dataset for mean comparisons; N=8 independent fine-tuning runs for correlation analyses), (iii) the significance threshold (α=0.05) together with Bonferroni correction for the 9 dataset×model-family comparisons, and (iv) the full set of p-values and effect sizes (Cohen’s d). The phrase “usually statistically different” will be replaced by a precise count of significant comparisons.","revision_made":"yes","referee_comment":"[Abstract / statistical results] Abstract and results on statistical comparisons: the claims that means and standard deviations 'are usually statistically different' and that there are 'some potential signals on overall positive correlation' lack any description of the tests performed, sample sizes per comparison, p-value thresholds, or corrections for multiple testing. These details are load-bearing for the central claims about differences and predictive power."},{"response":"We will add a dedicated subsection describing the procedure: last-token activations from the final layer are extracted as features; a ridge regression (α=1.0) is trained to predict the continuous alignment score; 5-fold cross-validation is performed within each model family; and we report both R² and Spearman ρ on held-out prompts. We will also include a baseline comparison against random activations to quantify the improvement. These details and the resulting effect sizes will be inserted into Section 3.2.","revision_made":"yes","referee_comment":"[Results on model priors / activation prediction] The strongest claim—that evaluation prompt-only activations from pre-trained and original instruct models predict fine-grained alignment scores after narrow fine-tuning—requires explicit reporting of the prediction procedure (feature extraction, regression or classifier used, cross-validation, effect sizes such as R² or AUC). Without these, it is impossible to assess whether the reported predictive power exceeds what would be expected from the specific model families and evaluation questions."},{"response":"We will clarify that subspace overlap is measured by the average cosine similarity between the top-8 principal components of the activation deltas (chosen via explained-variance elbow), that the correlation with shift similarity is Pearson r=0.67 (p<0.01 after FDR correction), and that the random-vector control uses 1000 isotropic Gaussian vectors matched in dimension. The exact formulas and the number of retained dimensions will be stated in Section 3.3 together with the reported values.","revision_made":"yes","referee_comment":"[Activation delta / subspace overlap results] Subspace overlap analysis: while a control against random vectors is mentioned, the manuscript should specify the exact metric for overlap (e.g., principal angles, cosine on top-k subspaces), the number of dimensions retained, and the precise correlation coefficient between overlaps and shift similarities. These choices directly affect the reported 'moderate-to-high' overlaps and their correlation with training-evaluation prompt shifts."}],"tokens_in":1550,"tokens_out":718,"duration_ms":16314,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper reports two main empirical observations from narrow fine-tuning runs: prompt-only activations taken before fine-tuning (from both base and instruct models) correlate with the alignment scores measured afterward, and the activation shifts for training prompts overlap moderately with those for evaluation prompts, with that overlap itself correlating to shift similarity at the last token. They also checked that training loss does not reliably predict out-of-domain alignment and that alternative learning schedules did not produce clearly better broad alignment at similar loss.\n\nThose links between pre-fine-tuning activations and later misalignment scores, plus the subspace overlap measurements with a random-vector control, are the concrete new pieces. The work is straightforward empirical tracking of internals across a few model families and datasets, which is useful for people already measuring activation geometry during alignment experiments.\n\nThe main limitation is that the abstract states statistical differences and correlations without naming the tests, sample sizes, or any multiple-comparison adjustments, so the strength of the reported signals is hard to gauge from the given text. Everything stays observational, so the predictive relationships could still be tied to the particular models, prompts, or fine-tuning setups rather than general features of EM. No causal claims are made, which keeps the scope honest but also limits how far the results travel.\n\nThis is for readers already working on LLM internals and alignment monitoring who want additional correlation data to test against their own runs. It is not yet strong enough to change practice on its own, but the observations are specific enough that a referee could usefully check the methods and ask for more controls. I would send it to review rather than desk-reject.","headline":"Pre-fine-tuning prompt activations predict post-fine-tuning alignment scores in their experiments, with moderate subspace overlap between train and eval shifts, but the statistical reporting is thin.","tokens_in":2309,"tokens_out":402,"would_cite":false,"duration_ms":17028,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Pre-fine-tuning activations predict fine-grained alignment scores after narrow fine-tuning.","keywords":["emergent misalignment","narrow fine-tuning","model activations","subspace overlap","training dynamics","alignment prediction","language models","model priors"],"falsifier":"Observing no correlation between pre-fine-tuning activations and post-fine-tuning alignment scores in a new model family or dataset would falsify the predictive claim.","tokens_in":2651,"feed_emoji":"","tokens_out":565,"duration_ms":18548,"temperature":0.7,"pith_summary":"The paper examines the factors shaping emergent misalignment, where narrow fine-tuning on specific data leads to broad misalignment on other tasks. It analyzes training dynamics by relating in-domain loss to out-of-domain scores, model priors through activation patterns before fine-tuning, and data via subspace overlaps in activation changes. The central result is that activations from pre-trained and instruct models before fine-tuning can predict the alignment scores after the fine-tuning process. This points to model priors as a key driver of how misalignment generalizes unevenly.","feed_headline":"Pre-fine-tuning activations predict post-training misalignment","feed_subtitle":"Signals in model activations before narrow fine-tuning forecast how much broad misalignment will emerge afterward.","key_machinery":"Prompt-only activations from models prior to narrow fine-tuning and the subspace overlaps between training and evaluation prompt activations.","core_discovery":"Evaluation prompt-only activations from pre-trained and original instruct models can predict fine-grained alignment scores after narrow fine-tuning, with moderate-to-high subspace overlap in activation shifts between training and evaluation prompts that correlates with their similarities.","pith_inferences":["If activations reliably predict misalignment risk, models could be screened for susceptibility before any fine-tuning occurs.","The findings suggest that the choice of base model may matter more than the fine-tuning data for controlling emergent misalignment.","Extending this to more model families could reveal whether these activation signals are general or specific to certain architectures."],"forward_implications":["In-domain training loss shows some positive correlation with out-of-domain misalignment but does not determine it strongly.","Different learning schedules for narrow fine-tuning do not produce runs with substantially better broad alignment at similar loss levels.","Activation deltas before and after fine-tuning exhibit moderate-to-high overlap between training and evaluation prompts.","Subspace overlaps between training and evaluation prompts correlate with similarity of their activation shifts."],"fun_headline_variants":["Pre-fine-tuning activations predict alignment scores","Subspace overlaps link training and evaluation shifts","Priors correlate with post-tuning misalignment","Prompt activations predict fine-grained alignment scores"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The observed predictive power of pre-fine-tuning activations and subspace overlaps is not limited to the specific model families, datasets, or evaluation questions used in the experiments.","fun_headline_variants_meta":{"raw":{"variants":["Pre-fine-tuning activations predict alignment scores","Subspace overlaps link training and evaluation shifts","Priors correlate with post-tuning misalignment","Prompt activations predict fine-grained alignment scores"]},"model":"grok-4.3","cost_usd":0.004462,"raw_usage":{"total_tokens":2220,"prompt_tokens":655,"num_sources_used":0,"completion_tokens":51,"cost_in_usd_ticks":44624500,"prompt_tokens_details":{"text_tokens":655,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1514,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":655,"tokens_out":51,"duration_ms":10859,"temperature":1.0,"reasoning_tokens":1514,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T17:17:05.246079+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Observing no correlation between pre-fine-tuning activations and post-fine-tuning alignment scores in a new model family or dataset would falsify the predictive claim.","supporting_citations":[],"review_version":1}